Lumberjack


The Lumberjack Camel component consumes logs from Filebeat. Learn how to use the Lumberjack component. 

Compatibility and support

Technically, the Lumberjack component implements versions 1 and 2 of the Lumberjack protocol.

However, differences exist in term of behavior and protocol implementation across the various Lumberjack client tools. Those differences require a specific validation for each such client tool, in order to handle to potential differences.

The following table describes the status for currently supported and unsupported Lumberjack client tools:

Client tool Protocol version used Communication supported Support on client tool
Filebeat 1.x.x (>= 1.2.3) 2 (tick) (tick) As part of the product, Axway provides support on the Filebeat client itself and its deployments. See Remote log streaming with Filebeat.
Filebeat 5.2.0 2

(tick)

(warning) The new Pipelining feature of Filebeat 5 is NOT supported yet

(tick) As part of the product, Axway provides support on the Filebeat client itself and its deployments. See Remote log streaming with Filebeat.
Logstash-forwarder 1 (error) Not validated, not supported, since Logstash-forwarder is deprecated.
Logstash 2.x(>= 2.1.3) 1 (tick) (error) Axway supports incoming communication from Logstash, but does not provide support on Logstash deployments and will consider them external data sources.
Logstash 5.x 1 (tick) (error) Axway supports incoming communication from Logstash, but does not provide support on Logstash deployments and will consider them external data sources.

Library

Create a new library with the following jars:

Camel version Library jars Supported lumberjack protocol versions
2.18.3 camel-lumberjack-1.1.0.jar version 1 and 2

See  How to retrieve the Camel version of Decision Insight.

Connectors

SslContextParameters

To configure the TLS, you need to create a connector according to How to configure SSL on a component.

Context

URI format

The URI format is lumberjack[s]:host[:port][?options]

Since Apache Camel 2.18.3 it's no longer needed to use lumberjacks in order to use SSL, you can use directly the lumberjack component.

URI samples Description
lumberjack:0.0.0.0 Listen on the default port (5044) on all network addresses
lumberjack:127.0.0.1:15044 Listen only on 15044 port on the loopback address only (will only accept connections from the same server)

lumberjack:0.0.0.0?sslContextParameters=#ssl

lumberjacks:0.0.0.0?sslContextParameters=#ssl

Listen for SSL encrypted communication on the default port (5044) on all network addresses and uses the connector named ssl for the SSL configuration.

URI options

Name Default value Description
sslContextParameters none

Its value is prefixed with a  # in order to indicate that it must be extracted from the Connectors

For security purpose, we highly recommend usage of TLS configuration.

Listen and parse logs

If you want to read such logs:

output.log
2016-05-17T14:37:29.160 [main] [INFO] org.apache.camel.impl.converter.DefaultTypeConverter - Loaded 182 type converters 

With TLS

With TLS encryption on port 15044 using the Decision Insight connector named sslContextParameters:

<routes xmlns="http://camel.apache.org/schema/spring" xmlns:u="http://www.systar.com/aluminium/camel-util">
    <route>
        <from uri="lumberjack:0.0.0.0:15044?sslContextParameters=#sslContextParameters"/>
        <!-- We parse the log message using a regular expression -->
        <setBody>
            <u:string-regexp
                    pattern="(?&lt;timestamp>\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}.\d{3}) \[(?&lt;thread>.+)] (?&lt;level>.+) (?&lt;category>.+) - (?&lt;message>.+)"
                    useNamedGroups="true">
                <simple>${body[message]}</simple>
            </u:string-regexp>
        </setBody>
        <log message="timestamp: ${body[timestamp]}, thread: ${body[thread]}, level: ${body[level]}, category: ${body[category]}, message: ${body[message]}"/>
    </route>
</routes>

Without TLS

TLS configuration

Although it is not mandatory, we advise you to always configure TLS connection and forbid non-encrypted communication. If not configured:

  • data are not encrypted
  • without mutual authentication, you cannot ensure that connected clients are trusted
Routes definition
<routes xmlns="http://camel.apache.org/schema/spring" xmlns:u="http://www.systar.com/aluminium/camel-util">
    <route>
        <from uri="lumberjack:0.0.0.0"/>
        <!-- We parse the log message using a regular expression -->
        <setBody>
            <u:string-regexp
                    pattern="(?&lt;timestamp>\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}.\d{3}) \[(?&lt;thread>.+)] (?&lt;level>.+) (?&lt;category>.+) - (?&lt;message>.+)"
                    useNamedGroups="true">
                <simple>${body[message]}</simple>
            </u:string-regexp>
        </setBody>
        <log message="timestamp: ${body[timestamp]}, thread: ${body[thread]}, level: ${body[level]}, category: ${body[category]}, message: ${body[message]}"/>
    </route>
</routes>

Manual configuration of the ACK

Default behavior

When the Camel processing is completed, if an exception was thrown and interrupted the flow of processing, then the automatic ACK is not sent.

In Camel you can manually:

  • throw a new exception using the <throwException> tag
  • catch thrown exceptions using the <doTry> and <doCatch> tags

Therefore, when combining those tags you can tune when to manually send an ACK.

Catch all errors

In this example, we're going to catch all errors and write them to disk as their original binary representation. Since we're catching the exception, it will automatically reply with ACK responses.

Route definition
<routes xmlns="http://camel.apache.org/schema/spring" xmlns:u="http://www.systar.com/aluminium/camel-util">
    <route>
        <from uri="lumberjack:0.0.0.0:15044?sslContextParameters=#sslContextParameters"/>
        <!-- We parse the log message using a regular expression -->
        <setBody>
            <u:string-regexp
                pattern="(?&lt;timestamp>\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}.\d{3}) \[(?&lt;thread>.+)] (?&lt;level>.+) (?&lt;category>.+) - (?&lt;message>.+)" useNamedGroups="true">
                <simple>${body[message]}</simple>
            </u:string-regexp>
        </setBody>
        <log message="timestamp: ${body[timestamp]}, thread: ${body[thread]}, level: ${body[level]}, category: ${body[category]}, message: ${body[message]}"/>
        <!-- Try to process the event -->
        <doTry>
            <to uri="direct:process"/>
            <doCatch>
                <!-- Catch all exceptions -->
                <exception>java.lang.Exception</exception>
                <!--Log the exception-->
                <log message="An exception has occured ${exception.message}"/>
            </doCatch>
        </doTry>
    </route>
    <route>
        <from uri="direct:process"/>
        <choice>
            <when>
                <simple>${body[level]} == '[ERROR]'</simple>
                <throwException
                    exceptionType="java.lang.Exception" message="An ERROR was found in the log"/>
            </when>
        </choice>
    </route>
</routes>

Related Links