Configure Filebeat

Learn how to configure Filebeat.

A full description of the YAML configuration file for Filebeat can be found in Filebeat 1.2 configuration options page or Filebeat 5.2.0 configuration options page.

Beware, the YAML syntax is very strict. For example tab characters are not allowed, your text editor may automatically use them for indentations but you will hardly notice it. Sometimes Filebeat reports the error, sometimes not. Verifying the configuration file with a YAML syntax checker might help.

Here is a basic sample:

filebeat.yml
filebeat:
  prospectors:
    - paths:
        - /home/ubuntu/somelogs/*.log  
  registry_file: .filebeat
output:
  logstash:
    hosts: ["192.168.1.225:5044"]

Input

Propectors

Prospectors are used to locate and process log files. Each prospector item begins with a dash (-) and specifies prospector-specific configuration options.

Parameter Default Mandatory Description
paths /var/log/*.log Yes

Specify the list of paths that are scanned to locate log files.

Each path item begins with a dash (-).

Example
  paths:
    # Parse all files with the extension .log directly under folder /home/ubuntu/somelogs
    - /home/ubuntu/somelogs/*.log
    # Parse all files with the extension .log under the first level of subdirectories of folder /home/ubuntu/somelogs
    - /home/ubuntu/somelogs/*/*.log
    # Parse all files with an extension prefixed by .log located directly under folder /home/ubuntu/somelogs
    - /home/ubuntu/somelogs/*.log*

Recursively matching all files in a directory and its subdirectories is currently not supported, but you can use a wildcard (*) for directory names. This means that you will have to declare a paths entry for each level of subdirectories you want to monitor.

Multiple prospectors

When using more than one prospector, you must ensure that each log file is monitored in the paths of only one prospector. Sharing the same file between multiple prospectors can lead to unexpected behaviour.

multiline Not set No

By default, Filebeat will treat each line in a log file as a separate log message.

When monitoring log messages that span multiple lines, you can use the multiline to group all lines of a message together following a pattern.

The most important parameters are:

Example
    (...)
        # Matches all lines starting with [
        pattern: ^\[
    (...)
        # Matches all lines starting with yyyy-MM-dd HH:mm:ss,SSS. Ex: 2016-05-17 12:09:52,702
        pattern: '^([0-9]{4})(-?)(1[0-2]|0[1-9])(-?)(3[01]|0[1-9]|[12][0-9])[[:space:]]([0-9]{2}):([0-9]{2}):([0-9]{2}),([0-9]{3})'
  • negate and match: Specifies how the pattern will be used to group multiple lines into a single message.

For more information, see configuration details.

include_lines All No

This is a comma separated list of regular expressions.

Log messages matching at least one of these expressions will be exported. Log messages that match none of these expressions are skipped.

By default, all log messages are exported.

Example
    (...)
        # Matches all lines starting with ERR
        include_lines: ["^ERR"]
    (...)
        # Matches all lines starting with ERR or WARN
        include_lines: ["^ERR", "^WARN"]
exclude_lines None No

This is a comma separated list of regular expressions.

Log messages matching at least one of these expressions will be skipped.

By default, no lines are skipped.

Example
    (...)
        # Matches all lines starting with INFO
        exclude_lines: ["^INFO"]
    (...)
        # Matches all lines starting with DEBUG or INFO
        exclude_lines: ["^DEBUG", "^INFO"]

When both include_lines and exclude_lines are defined, lines are filtered by include_lines first and then by exclude_lines.

 

Registry file

registry_file parameter is used to specify the registry file, used to keep the track of the already processed logs. Removing this file will clear the registry and log file parsing will restart.

 

Output

logstash output is a consumer to which Filebeat sends data using the Lumberjack protocol. ADI is such a consumer and communication with it will be defined as follows:

output:
  logstash:
    (...)
  • Output parameters for Filebeat version 1.2 :
Parameter Mandatory Description
hosts Yes

A comma-separated list of lumberjack listeners. The format is host:port.

Ex:  hosts: ["81.100.100.1:5044"]

To configure and start a Lumberjack listener in ADI, follow the steps from the Camel Lumberjack component page.

tls No

TLS communication protocol. See how to configure output tls documentation.

More output parameters are detailed in the output configuration options page.

  • Output parameters for Filebeat version 5.2.0 :
Parameter Mandatory Description
hosts Yes

A comma-separated list of lumberjack listeners. The format is host:port.

Ex:  hosts: ["81.100.100.1:5044"]

To configure and start a Lumberjack listener in ADI, follow the steps from the Camel Lumberjack component page.

ssl No TLS communication protocol. See how to configure output ssl documentation.

More output parameters are detailed in the output configuration options page

Recommendations

Rolling files

For optimal behavior regarding rolling files those parameter sould be false:

  • tail_files,
  • force_close_files for Filebeat v1.2 or close_removed for Filebeat v5.2.0.

To ensure that no line remain unprocessed upon file renaming, the new file name must be monitored in the prospector paths.

Logs scan rhythm

In the default configuration, a prospector will detect a new file within 10 seconds after it is actually created. The prospector will also detect a new line added to a known file within the next second after it is actually added. These settings can be adjusted with the scan_frequency and backoff parameters.

Path definition

Absolute and relative paths can be used, but absolute paths are more straightforward: relative paths correspond to the working directory. When restarting, changing the working directory will mean that the previous registry file won't be used and log parsing will be restarted.

Stop monitoring files based on their last modification time

The more files you are monitoring, the bigger your registry_file will become. To optimize its use, you should adjust the values of close_older (Filebeat v1.2) or close_inactive (Filebeat v5.2.0) and ignore_older  to suit the lifetime of your log files. In the default configuration, files will never be ignored and handlers on those that haven't been updated in more than 1h will be closed.

Compression

By default, Filebeat will compress all outgoing messages. In a situation where network bandwitdh is not a bottleneck and a minimum impact on the CPU usage of the monitored host is required, compression can be deactivated by setting compression_level to 0.

Related Links