How to parse fixed width and delimited files using Flatpack

Learn how to use the Flatpack component.

This example shows how to parse fixed width and delimited files using Camel and Flatpack.

For more details on using this library, refer to:

Library

To install the camel-flatpack library,  use the managed libraries repository.

  camel-flatpack can also be installed manually

Create a new library with the following jars (in this order):

Camel version Library jars
2.25.1

See How to retrieve the Camel version of Decision Insight.

Fixed width parser

Resource

Build your Flatpack parsing configuration file:

parser.pzmap.xml 
<?xml version='1.0'?>
<!DOCTYPE PZMAP SYSTEM "flatpack.dtd" >
<PZMAP>
    <COLUMN name="Sender" length="2" />
    <COLUMN name="Receiver" length="2" />
    <COLUMN name="SenderApp" length="3" />
    <COLUMN name="ReceiverApp" length="3" />
    <COLUMN name="Id" length="5" />
    <COLUMN name="ProcessedTime" length="12" />
    <COLUMN name="AcquiredTime" length="12" />
</PZMAP>


Upload the configuration file as a resource named parser.pzmap.xml:

Connector

The connector is used to instantiate and configure the parser.

Create a new connector, named flatpack:

  • Class type: other
  • Class name: org.apache.camel.dataformat.flatpack.FlatpackDataFormat
    • in library flatpack
  • Configure properties:
    • definition: tnd-resource/parser.pzmap.xml
    • ignoreExtraColumns: true
    • ignoreFirstRecord: false
    • fixed: true

Route

Create the following sample route, that will simply:

  • Read the input
  • Parse it using flatpack (e.g. unmarshal it). Result is in body with type List<Map<String, String>>
  • Iterate through results
    • Log each parsed result (of type Map<String, String>)
routes.xml 
<routes xmlns:u="http://www.systar.com/aluminium/camel-util" xmlns="http://camel.apache.org/schema/spring">
    <route>
        <from uri="direct:foo"/>
        <unmarshal>
            <custom ref="flatpack"/>
        </unmarshal>
        <split>
            <simple>${body}</simple>
            <to uri="log:camel-flatpack"/>
        </split>
    </route>
</routes> 

Once the route is configured, start the route.

Sample file

Save the following sample file as sample.txt:

sample.txt 
0065ABCEFG00025201301240639            E
0000ABCEFH00000201301240838            E
0061ABDEFG00035201301241044            E
0064ABCEFG00036201301241044            E
0032ABCEFG00032201301241049            E
0000ABCEFG00000201201241637            E


Upload and inject the sample file manually as input for the flatpack route:

Results

2013-02-01 00:00:22,045 [1730271140@qtp-267898584-36] INFO camel-flatpack - Exchange[ExchangePattern:InOnly, BodyType:java.util.HashMap, Body:{Sender=00, SenderApp=ABC, ReceiverApp=EFG, Receiver=65, AcquiredTime=, Id=00025, ProcessedTime=201301240639}]
2013-02-01 00:00:22,088 [1730271140@qtp-267898584-36] INFO camel-flatpack - Exchange[ExchangePattern:InOnly, BodyType:java.util.HashMap, Body:{Sender=00, SenderApp=ABC, ReceiverApp=EFH, Receiver=00, AcquiredTime=201301240838, Id=00000, ProcessedTime=}]
2013-02-01 00:00:22,091 [1730271140@qtp-267898584-36] INFO camel-flatpack - Exchange[ExchangePattern:InOnly, BodyType:java.util.HashMap, Body:{Sender=00, SenderApp=ABD, ReceiverApp=EFG, Receiver=61, AcquiredTime=, Id=00035, ProcessedTime=201301241044}]
2013-02-01 00:00:22,092 [1730271140@qtp-267898584-36] INFO camel-flatpack - Exchange[ExchangePattern:InOnly, BodyType:java.util.HashMap, Body:{Sender=00, SenderApp=ABC, ReceiverApp=EFG, Receiver=64, AcquiredTime=, Id=00036, ProcessedTime=201301241044}]
2013-02-01 00:00:22,093 [1730271140@qtp-267898584-36] INFO camel-flatpack - Exchange[ExchangePattern:InOnly, BodyType:java.util.HashMap, Body:{Sender=00, SenderApp=ABC, ReceiverApp=EFG, Receiver=32, AcquiredTime=, Id=00032, ProcessedTime=201301241049}]
2013-02-01 00:00:22,094 [1730271140@qtp-267898584-36] INFO camel-flatpack - Exchange[ExchangePattern:InOnly, BodyType:java.util.HashMap, Body:{Sender=00, SenderApp=ABC, ReceiverApp=EFG, Receiver=00, AcquiredTime=201201241637, Id=00000, ProcessedTime=}]

Alternative: Delimited parser

Flatpack can also parse CSV, with more advanced capabilities than the basic Camel csv component. The output is of type List<Map<String, String>>, which is more convenient and less error-prone to use.

Here is an example with automated parsing, without explicit configuration file, that uses the CSV header as column names in the output map. The protocol is the same as for delimited parsing, with the following content instead for the different steps.

Connector

Create a new connector, named flatpack_csv:

  • Class type: other
  • Class name: org.apache.camel.dataformat.flatpack.FlatpackDataFormat 
    • in library flatpack
  • Configure properties:
    • ignoreFirstRecord: true (If the first line contains columns meta-data)
    • fixed: false

Route

routes.xml 
<routes xmlns:u="http://www.systar.com/aluminium/camel-util" xmlns="http://camel.apache.org/schema/spring">
    <route>
        <from uri="direct:foo"/>
        <unmarshal>
            <custom ref="flatpack_csv"/>
        </unmarshal>
        <split>
            <simple>${body}</simple>
            <to uri="log:camel-flatpack"/>
        </split>
    </route>
</routes> 

Sample CSV

sample.csv 
"FIRSTNAME","LASTNAME","ADDRESS","CITY","STATE","ZIP"
"JOHN","ANAME","1234 CIRCLE CT","ELYRIA","OH","44035"
"JIMMY","ZNAME","180 SOME ST","AVON","OH","44011"
"JANE","ANAME","111 MILKY WY","AVON","OH","44001"
"FRED","ZNAME","123 ROCKY WY","ELYRIA","OH","12345"
"FRED","ZNAME","123 ROCKY WY","ELYRIA","OH","12345"
"FRED","ZNAME","Text Qualifier " and separator, in string","ELYRIA","OH","" 

Results for CSV

2013-02-01 00:09:10,738 [184058750@qtp-267898584-32] INFO camel-flatpack - Exchange[ExchangePattern:InOnly, BodyType:java.util.HashMap, Body:{FIRSTNAME=JOHN, STATE=OH, ZIP=44035, ADDRESS=1234 CIRCLE CT, CITY=ELYRIA, LASTNAME=ANAME}]
2013-02-01 00:09:10,739 [184058750@qtp-267898584-32] INFO camel-flatpack - Exchange[ExchangePattern:InOnly, BodyType:java.util.HashMap, Body:{FIRSTNAME=JIMMY, STATE=OH, ZIP=44011, ADDRESS=180 SOME ST, CITY=AVON, LASTNAME=ZNAME}]
2013-02-01 00:09:10,739 [184058750@qtp-267898584-32] INFO camel-flatpack - Exchange[ExchangePattern:InOnly, BodyType:java.util.HashMap, Body:{FIRSTNAME=JANE, STATE=OH, ZIP=44001, ADDRESS=111 MILKY WY, CITY=AVON, LASTNAME=ANAME}]
2013-02-01 00:09:10,740 [184058750@qtp-267898584-32] INFO camel-flatpack - Exchange[ExchangePattern:InOnly, BodyType:java.util.HashMap, Body:{FIRSTNAME=FRED, STATE=OH, ZIP=12345, ADDRESS=123 ROCKY WY, CITY=ELYRIA, LASTNAME=ZNAME}]
2013-02-01 00:09:10,741 [184058750@qtp-267898584-32] INFO camel-flatpack - Exchange[ExchangePattern:InOnly, BodyType:java.util.HashMap, Body:{FIRSTNAME=FRED, STATE=OH, ZIP=12345, ADDRESS=123 ROCKY WY, CITY=ELYRIA, LASTNAME=ZNAME}]
2013-02-01 00:09:10,741 [184058750@qtp-267898584-32] INFO camel-flatpack - Exchange[ExchangePattern:InOnly, BodyType:java.util.HashMap, Body:{FIRSTNAME=FRED, STATE=OH, ZIP=, ADDRESS=Text Qualifier " and separator, in string, CITY=ELYRIA, LASTNAME=ZNAME}]

Related Links