Tuesday, May 24, 2011

Reading Delimited Files Using FlatPack

FlatPack is a java tool box which supports you to write & read delimited files. This is a sample java code that i have tried out using FlatPack to delimited my .txt file which contains the columns names as the first record.

This sample code will describe you how to insert the content of a delimited file into a database. I have tried out my code in netbeans. You should import all the dependency jar files in to your project (to lib folder) before you run this code. Download dependency files flatpack.jar, jdom.jar & slf4j-api.jar.

Sample .txt file (test.txt)
"JOHN","ANAME","1234 CIRCLE CT","ELYRIA","OH","44035"
"JIMMY","ZNAME","180 SOME ST","AVON","OH","44011"
"JANE","ANAME","111 MILKY WY","AVON","OH","44001"
"FRED","ZNAME","123 ROCKY WY","ELYRIA","OH","12345"
"FRED","ZNAME","123 ROCKY WY","ELYRIA","OH","12345"

FileReader.java file
import java.io.File;
import net.sf.flatpack.DataSet;
import net.sf.flatpack.DefaultParserFactory;
import net.sf.flatpack.Parser;

import java.sql.Connection;
import java.sql.DriverManager;
import java.sql.PreparedStatement;
import java.sql.Statement;
import java.sql.ResultSet;

public class FileReader {

    static Connection con;

    public static void main(String args[]) throws Exception {

        con = DriverManager.getConnection("jdbc:derby:myDB;create=true;user=admin;password=admin");//enter details of your database
        Statement stmt = con.createStatement();
        //create a new table in "myDB" database
        stmt.executeUpdate("CREATE TABLE USER_NAMES (first_name VARCHAR(32) NOT NULL, last_name VARCHAR(32) NOT NULL)");

        //.txt file type used here, comma delimted with text qualified by "'s
        Parser pzparser = DefaultParserFactory.getInstance().
                newDelimitedParser(new File("test.txt"), ',', '"');
        final DataSet ds = pzparser.parse();

        while (ds.next()) {
            //insert values into database by using the column names of the .txt file
            PreparedStatement psInsert = con.prepareStatement("insert into USER_NAMES values (?,?)");
            psInsert.setString(1, ds.getString("FIRSTNAME"));
            psInsert.setString(2, ds.getString("LASTNAME"));
If you need to print records inside the "USER_NAMES" table, you just need to copy following code snippet after the while loop.
Statement ss = con.createStatement();
        ResultSet rs = ss.executeQuery("select * from USERS");
        System.out.println("First Name | Last Name ");
        while (rs.next()) {
            System.out.println(rs.getString(1) + "\t" + rs.getString(2));

Hope this post will be a useful one for you all. :)

Wednesday, May 18, 2011

UltraESB File Handling Sample

It is a problem that someone have to continuously monitor a folder or group of folders for new files then trigger a custom action such as categorizing them, convert into desired format or response them etc

So this article demonstrates the capability of the Free and  Open Source UltraESB to pick up a file from a directory, process it within the ESB and finally move the files to corresponding directories after processing, depending on the outcome.

If you installed the UltraESB v1.4.0 runtime version you’ll find all available examples inside the “ultraesb-1.4.x/samples/conf ” directory
( /home/imalka/java/ultraesb-1.4.0/samples/conf  with respect to mine)

Today I am going to work with the sample-401 (ultra-sample-401.xml configuration file) . To try out this sample you should first create the directory structure to store the files that need to be processed and the files after processing

According to the configuration, my directory hierarchy is created in the “/tmp” directory in linux operating system. The top level directory is named as “file”. One can create this folder structure in any location on the computer where you have permissions to update (instead of /tmp directory) 

Inside “/tmp/file” directory

incoming - the directory which contains all the files which needs to be processed
To store processed files,
error - contains failed files after processing
done - contains successfully processed files
sent  - we are not discuss about this directory in this article. But you can find an  example related to this in sample-401(in file-proxy5)

Running the sample
Executing configuration

Then start the UltraESB configuration, execute sample 401 as follows.
In my case,
imalka@imalka:~/java/ultraesb-1.4.0/bin$ ./ultraesb.sh -sample 401                     

Insert files into “ /tmp/file/incoming ” diectory

Now you can create files according to the configurations define in sample 401.


<u:proxy id="file-proxy1">
        <u:transport id="file">
            <u:property name="url" value="file:///tmp/file/incoming"/>
            <u:property name="fileNamePattern" value=".*\.txt"/>
            <u:property name="startDelay" value="1000"/>
            <u:property name="repeatInterval" value="2000"/>
            <u:property name="moveAfterProcess" value="/tmp/file/done"/>
            <u:property name="moveAfterFailure" value="/tmp/file/error"/>
            <u:property name= "moveTimestampFormat" value="yyyy_MM_dd_'T'HH_mm_ss.SSSSZ"/>
                    String fileName = msg.getFirstTransportHeader("FileName");
                    System.out.println("Got file : " + fileName);
                    if ("error.txt".equals(fileName)) {
                        throw new Exception("This file should fail");

The sample below shows the configuration used in the example, which can be easily modified and extended. The example configuration we use is as follows. it uses the method msg.getFirstTransportHeader() API by passing corresponding file name as a String.

This code snippet in the configuration (UltraESB sample # 401) defines a proxy service that start up after 1 second and polls for "*.txt" files at /tmp/file/incoming directory every 2 seconds. Processed and failed files are moved to /tmp/file/done or /tmp/file/error directories with a time stamp.  This proxy throws an exception if the filename patches 'error.txt' so that the received files maybe tested.

Creating files for testing
Create two files, “test.txt” & “error.txt” inside “/tmp/file/incoming ” directory. After processing those two files “test.txt” file will be moved into “/tmp/file/done” directory and “error.txt” file be classified as failed file because of its name and move into “ /tmp/file/error ” directory.


To try out this you should first start a test server to simulate a web service 
The UltraESB ships a sample Jetty server with web services one could use for testing. This could be started using the ToolBox as follows,

imalka@imalka:~$ cd /home/imalka/java/ultraesb-1.4.0/bin/

Now AdroitLogic toolbox for Ultraesb window will be appear in your work place.
Go to file menu  -->  new  -->  Jetty Server (or Ctrl + j)
Start Jetty (in port 9000)

To get a better idea of  AdroitLogic ToolBox for the UltraESB refer the following link  
<u:proxy id="file-proxy3">
        <u:transport id="file">
            <u:property name="url" value="file:///tmp/file/incoming/response.test"/>
            <u:property name="startDelay" value="1000"/>
            <u:property name="repeatInterval" value="2000"/>
            <u:property name="moveAfterProcess" value="/tmp/file/done"/>
            <u:property name="moveAfterFailure" value="/tmp/file/error"/>
            <u:property name="moveTimestampFormat" value="yyyy_MM_dd_'T'HH_mm_ss.SSSSZ"/>
                    String fileName = msg.getFirstTransportHeader("FileName");
                    System.out.println("Got file : " + fileName);
                    if ("response".equals(mediation.readPayloadAsString(msg))) {
                        System.out.println("Expected response received");
                    } else {
                        throw new Exception("This file should fail");

The above code snippet defines a proxy that polls /tmp/file/incoming/response.test every 2 seconds and holds off message completion by using the method msg.holdCompletion() API until the file payload is sent to an HTTP endpoint at http://localhost:9000/service/EchoService and a response is received. Success or failure in file processing is depend on the result of server response.

Creating files for testing
Create two files and name them as, “ response.test ” (inside “/tmp/file/incoming ” directory) and type “response” and “test”.
When processing two files poxy service searching for the expected results (the content “response”) inside the files. 
 After processing only the file which contains the expected content will be moved into “/tmp/file/done” directory and the other file will be moved into “ /tmp/file/error ” directory.

Note :
Instead of throwing exception you can use setMarkedAsFailed() method in Message class.
Just replace,
throw new Exception("This file should fail");
line in “ ultra-sample-401.xml ” cofiguration file with,
This method mark the message as failed, by passing true. For more details on that method please refer  the Javadoc API .

Friday, May 6, 2011

Quick Intro to UltraESB

Why UltraESB

The UltraESB is very easy to use and light weight! Only 35MB to download including ready to run samples


Operating System

The UltraESB works best on a Linux operating system including best performance, and ease of configuration and tuning of the OS but it also work perfectly on windows system(Because UltraESB is a pure Java program)

Java Development Kit

The UltraESB is currently certified against a JDK 1.6.x Installing Java on Ubuntu

You just have to follow simple steps to complete the installation

Step 1 - Download UltraESB

you can download latest UltraESB runtime version as a zip file from the following link to AdroitLogic website Download now

Step 2 – Unzip the adroitlogic-ultraesb-commercial-1.4.x-bin.zip

Here I extract the zip file to “ /home/imalka/java “ directory

imalka@imalka:~$ cd /home/imalka/java/

imalka@imalka:~/java$ unzip /home/imalka/Downloads/adroitlogic-ultraesb-commercial-1.4.0-bin.zip

The directory structure inside “ home/imalka/java/ultraesb-1.4.x”