Filebeat and Decision Insight usage

Start listening for messages

Filebeat uses the Lumberjack protocol to send messages to a listener in ADI.

Setup a listener using the Camel Lumberjack component to start receiving messages from Filebeat. 

Understanding the Lumberjack exchange content

Here is the sample definition of the exchange based on the JSON data sent by Filebeat. The content might change depending on your Filebeat version and options.

Body content
{
	"@metadata": {
		"type": "log",
		"beat": "filebeat"
	},
	"@timestamp": "2016-05-17T10:05:04.196Z",
	"beat": {
		"hostname": "pcqax1",
		"name": "pcqax1"	
	},
	"fields": "null",
	"input_type": "log",
	"message": "2016-04-29 13:50:46,526 [thread #5] INFO component - Component is started",
	"source": "/path/to/file.log",
	"type": "log"
}
Line Description
11

A sub-dictionary of custom fields defined in the Filebeat prospector that sent the message.

To store the custom fields as top-level fields (at the same level as input_type, message, ...), set the fields under root option to true.

(warning) When custom fields are stored under root, if a field name conflicts with a field name added by Filebeat, the custom field takes precedence, overriding the information from Filebeat.

12 The input_type set for Filebeat, describing the source type of the message.
13 The actual message body collected by Filebeat.
14 The source from which the message was collected.
15 The document_type option set on the harvesting prospector.


Ensure message resiliency

For reliable information that is never lost - not even at product or server crash - you must implement a resilient way of processing received messages.

The recommended implementation is detailed under How to ensure message resiliency.



Related Links