Install and configure Filebeat


Filebeat is the Axway supported log streamer used to communicate transaction and system events from an API Gateway to the ADI Collect Node. You must install and configure Filebeat once on each node where API Gateway is installed.

Filebeat is able to monitor all the events logs recorded by all the API Gateway instances installed on a single node. It connects to the ADI Collect Node and pushes each event that is recorded by API Gateway.

The following sample architecture diagram shows 2 API Gateway nodes, with 1 API Gateway instances each. On each Gateway instance one single Filebeat installation monitors all the event logs and streams the events to a single ADI instance:

Deploy Filebeat on Windows

Requirements

Filebeat must be installed and configured. Normally, filebeat is shipped with the Gateway and it is installed in  [api gateway install dir]/tools/filebeat-5.2.0.

Configure automatic start of Filebeat at boot and automatic respawn after crash

  • Open a windows PowerShell prompt as administrator (windows key > type PowerShell > right click > Run as Administrator)
  • Enter the following commands :
cd c:\Axway\filebeat-5.2.0-windows-x86_64
PowerShell.exe -ExecutionPolicy UnRestricted -File .\install-service-filebeat.ps1
  • After properly configuring c:\Axway\filebeat-5.2.0-windows-x86_64\filebeat.yml (see paragraph below), start Filebeat from the services console (Windows Key + R > type services.msc > enter > right-click on the filebeat service > start)

Deploy Filebeat on Linux

Requirements

Filebeat must be installed and configured. Normally, filebeat is shipped with the Gateway and it is installed in [api gateway install dir]/tools/filebeat-5.2.0.

Configure automatic start of Filebeat at boot and automatic respawn after crash

Depending on the version of Linux where API Gateway is running, three situations are possible: 

  • System V is the init system (Red Hat Enterprise Linux 5, SuSe Linux 11, CentOS 5, Oracle Linux 5, Debian 5) 
  • Upstart is the init system (Red Hat Enterprise Linux 6, Oracle Linux 6, Ubuntu <= 14)
  • systemd is the init system (Red Hat Enterprise Linux 7, Orcle Linux 7, SuSE Linux 12, Ubuntu 15)

Linux running System V

Automatic start at boot

As root, create the file  /etc/init.d/filebeat  with the following content: 

/etc/init.d/filebeat
#!/bin/sh
### BEGIN INIT INFO
# Provides:          filebeat
# Required-Start:    $local_fs $network $named $time $syslog
# Required-Stop:     $local_fs $network $named $time $syslog
# Default-Start:     3 2 5 4
# Default-Stop:      0 1 6
# Short-Description: API Gateway to ADI communication
# Description:       filebeat is a shipper part of the Elastic Beats
#                    family. Please see: https://www.elastic.co/products/beats
### END INIT INFO
WORKDIR="[api gateway install dir]/tools/filebeat-5.2.0"
PID="$WORKDIR/filebeat.pid"
USER=axway
NAME=filebeat
CMD="$WORKDIR/filebeat"
ARGS="-c $WORKDIR/filebeat.yml"

## Get the proper function library.
if [ -f /lib/lsb/init-functions ]; then
  . /lib/lsb/init-functions
elif [ -f /etc/init.d/functions ] ; then
  . /etc/init.d/functions
elif [ -f /etc/rc.d/init.d/functions ] ; then
  . /etc/rc.d/init.d/functions
fi

do_start() {
  echo "Starting $NAME"
  if [ -f "$PID" ] && kill -0 $(cat "$PID") 2>/dev/null && [ $(ps -p $(cat "$PID") | grep $NAME | wc -l) -eq 1 ]  2>/dev/null; then
    echo "Service already running" >&2
    return 1
  else
    su --login $USER --shell /bin/sh --command "cd $WORKDIR && ($CMD $ARGS >/dev/null 2>&1 &)"
    sleep 2
    ps -e -o pid,args |grep $CMD |grep -v grep|awk '{ print $1 }' > $PID
    return 0
  fi
}
do_stop() {
  if [ -f "$PID" ] && kill -0 $(cat "$PID") 2> /dev/null; then
    echo "Stopping $NAME"
    return $(killproc -p "$PID" "$CMD")
  else
    echo "Service $NAME is not running" >&2
    return 0
  fi
}
do_restart() {
  local result
  do_stop
  result=$?
  if [ $result -eq 0 ]; then
    sleep 2
    do_start
    result=$?
  fi
  return $result
}
do_status() {
  if [ -f "$PID" ] && kill -0 $(cat "$PID") 2> /dev/null; then
    echo "Service $NAME is running"
    return 0
  else
    echo "Service $NAME is not running"
    return 1
  fi
}
do_usage() {
  echo $"Usage: $0 {start | stop | restart | status}"
  exit 1
}
case "$1" in
start)   do_start;   exit $? ;;
stop)    do_stop;    exit $? ;;
restart) do_restart; exit $? ;;
status)  do_status;  exit $? ;;
*)       do_usage;   exit  1 ;;
esac

Next, execute the following command: 

chmod 755 /etc/init.d/filebeat

Next, depending on the Linux version, execute: 

  • On Debian or Ubuntu: 
update-rc.d filebeat enable
  • On Red Hat Enterprise Linux, CentOS, Oracle Linux, SuSE:
chkconfig --add filebeat
Automatic respawn

Add a respawn line at the bottom of  /etc/inittab  to allow automatic respawn of Filebeat after a crash:

fb:2345:respawn:/etc/init.d/filebeat start
Start and stop Filebeat manually

Run the following commands as root:

  • To start Filebeat
/etc/init.d/filebeat start
  • To check Filebeat status
/etc/init.d/filebeat status
  • To stop Filebeat and prevent an automatic respawn
    • Comment the line in  /etc/inittab
#fb:2345:respawn:/etc/init.d/filebeat start
    • Inform the init process that its configuration has changed
kill -HUP pid-of-init
    • Stop the Filebeat process
/etc/init.d/filebeat stop

Linux running Upstart

Automatic start at boot and automatic respawn

As root user, create the file  /etc/init/filebeat.conf  with the following content:

/etc/init/filebeat.conf
# filebeat - API Gateway to ADI communication
#
# filebeat is a shipper part of the Elastic Beats family. Please see:
# https://www.elastic.co/products/beats

description     "filebeat - API Gateway to ADI communication"
start on runlevel [2345]
stop on runlevel [!2345]

respawn
respawn limit 2 60

chdir /opt/Axway/filebeat
exec su -s /bin/sh -c 'exec "$0" "$@"' axway -- /opt/Axway/filebeat/filebeat -c [api gateway install dir]/tools/filebeat-5.2.0/filebeat.yml >/dev/null 2>&1

Next, execute:

chmod 644 /etc/init/filebeat.conf
Start and stop Filebeat manually

Run the following commands as root: 

  • To start Filebeat
initctl start filebeat
  • To check Filebeat status
initctl status filebeat
  •  To stop Filebeat
initctl stop filebeat

Linux running Systemd

Automatic start at boot and automatic respawn

As root user, create the file   /etc/systemd/system/filebeat.service  with the following content:

/etc/systemd/system/filebeat.service
[Unit]
Description=filebeat - API Gateway to ADI communication
After=network.target

[Service]
ExecStart=[api gateway install dir]/tools/filebeat-5.2.0/filebeat -c [api gateway install dir]/tools/filebeat-5.2.0/filebeat.yml
KillMode=process
User=axway
WorkingDirectory=[api gateway install dir]/tools/filebeat-5.2.0
Restart=always
RestartSec=30s

[Install]
WantedBy=multi-user.target

Next, execute the following commands:

chmod 644 /etc/systemd/system/filebeat.service
systemctl daemon-reload
systemctl enable filebeat.service
Start and stop Filebeat manually

Run the following commands as root: 

  • To start Filebeat
systemctl start filebeat.service
  • To check Filebeat status
systemctl status filebeat.service
  •  To stop Filebeat
systemctl stop filebeat.service

Configure filebeat.yml

Edit the file [filebeat install dir]/filebeat.yml . The content of the file should be similar to the example below.

With this sample configuration : 

  • Filebeat monitors two API gateway instances that are running on a single host.
  • Filebeat opens TCP connections on the ADI node on port 5044.
  • The connection on the ADI node on port 5044 is secured by TLSv1.2 mutual authentication. For more information see How to generate keys and certificates files for TLS mutual authentication?. Filebeat requires keys and certificates in OpenSSL format.
  • Filebeat is able to detect a newly created file within 10 seconds after it is created by API Gateway.
  • A newly recorded event is detected and sent to ADI within 1 second after it is added to a monitored event log file.
  • Event files older than 24 hours are not monitored by Filebeat.


filebeat.yml for Filebeat v1 to v5
filebeat:
  prospectors:
      # configure monitoring for first instance
    - paths:
          # monitor the event log file currently being written by API GW
        - [api gateway install dir]/events/group-1_instance-1*
          # monitor the event log files that the API GW node manager has already processed
          #  Used if API Analytics is running simultaneously with Embedded Analytics 
        - [api gateway install dir]/events/processed/group-1_instance-1_*.log.PROCESSED
      # new files are detected within 10s after they're created
      scan_frequency: 10s
      # new lines in a known file are detected within 1s afer they're written
      backoff: 1s
      # files are closed after 1 minutes without activity
      close_inactive: 1m
      # files older than 24h are ignored
      ignore_older: 24h
      # clean entries files older than 2 days of the registry file 
      clean_inactive: 48h
      fields:
        # group matches the group id in the monitored files
        group: 1
        # instance matches the instance id in the monitored files
        instance: 1
        # Front-End since this instance is accessed directly by end users
        type: Front-End
      # configure monitoring for second instance
    - paths:
          # monitor the event log file currently being written by API GW
        - [api gateway install dir]/events/group-1_instance-2*
          # monitor the event log files that the API GW node manager has already processed
          #  Used if API Analytics is running simultaneously with Embedded Analytics 
        - [api gateway install dir]/events/processed/group-1_instance-2_*.log.PROCESSED
      # new files are detected within 10s after they're created
      scan_frequency: 10s
      # new lines in a known file are detected within 1s afer they're written
      backoff: 1s
      # files are closed after 1 minutes without activity
      close_inactive: 1m
      # files older than 24h are ignored
      ignore_older: 24h
      # clean entries files older than 2 days of the registry file 
      clean_inactive: 48h
      fields:
        # group matches the group id in the monitored files
        group: 1
        # instance matches the instance id in the monitored files
        instance: 2
        # Back-End since this instance is accessed directly by end users
        type: Back-End
  # in this file filebeat remembers its current position in each monitored file 
  registry_file: [filebeat install dir]/.filebeat
output:
  logstash:
    # replace with the IP address of ADI
    hosts: ["adi.collector.node.ip:5044"]
    # filebeat will not compress data sent to ADI
    compression_level: 0
    # Number of workers per Logstash host.
    workers: 4
    ssl:
      # List of root certificates for server verifications
      certificate_authorities: ["[filebeat install dir]/myCA.pem"]
      # client certificate + key for client auth (if requested by server)
      certificate: "[filebeat install dir]/cert.pem"
      # key encrypted in aes-256-cbc
      key: "[filebeat install dir]/certkey.pem"
      key_passphrase: "axway*"
      # List of supported/valid TLS versions. 
      supported_protocols: [TLSv1.2]
      # Configure cipher suites to be used for SSL connections
      cipher_suites: [ECDHE-RSA-AES-256-GCM-SHA384]
logging:
  level: warning
  to_files: true
  to_syslog: false
  files:
    # configure the place where filebeat logs will be put. Make sure that the us er that will
    #    run the filebeat process will have the adapted rights on this directory
    path: [logging-directory]
    name: filebeat.log
    keepfiles: 7
filebeat.yml for Filebeat v5
##################$$$### Filebeat Configuration SAMPLE ########################

# This file is a configuration example to send API Gateway logs to Decision Insight
# 
# With this sample configuration : 
#   -  Filebeat monitors two API gateway instances that are running on a single host.
#   -  Filebeat opens TCP connections on the ADI node on port 5044.
#   -  The connection on the ADI node on port 5044 is secured by TLSv1.2 mutual
# authentication. Filebeat requires keys and certificates in OpenSSL format.
#   -  Filebeat is able to detect a newly created file within 10 seconds after
# it is created by API Gateway.
#   -  A newly recorded event is detected and sent to ADI within 1 second after
# it is added to a monitored event log file.
#   -  Event files older than 24 hours are not monitored by Filebeat.
#
# You can find the full configuration reference here:
# https://techweb.axway.com/public/display/DOCEAAPI/Install+and+configure+Filebeat

#=========================== Filebeat prospectors =============================

# List of prospectors to fetch data.
filebeat.prospectors:
# Each - is a prospector. Most options can be set at the prospector level, so
# you can use different prospectors for various configurations.
# Below are two prospectors specific configurations for :
#   - one Front-End API Gateway instance.
#   - one Back-End API Gateway instance.

#----------------------- Front-End instance prospector -------------------------
- input_type: log
# configure monitoring for first Front-End instance prospector
  paths:
    # monitor the event log file currently being written by API GW
    # --> REPLACE path to api gateway logs
    - [api gateway install dir]/events/group-1_instance-1*
    # monitor the event log files that the API GW node manager has already processed
    #  Used if API Analytics is running simultaneously with Embedded Analytics
    # --> REPLACE path to api gateway logs or remove it if there is no API Analytics
    - [api gateway install dir]/events/processed/group-1_instance-1_*.log.PROCESSED
  # new files are detected within 10s after they're created
  scan_frequency: 10s
  # new lines in a known file are detected within 1s afer they're written
  backoff: 1s
  # files are closed after 1 minutes without activity
  close_inactive: 1m
  # files older than 24h are ignored
  ignore_older: 24h
  # clean entries files older than 2 days of the registry file
  clean_inactive: 48h
  fields:
    # group matches the group id in the monitored files
    group: 1
    # instance matches the instance id in the monitored files
    instance: 1
    # Front-End since this instance is accessed directly by end users
    type: Front-End

#------------------------ Back-End instance prospector -------------------------
# configure monitoring for second Back-End instance prospector
- input_type: log
  paths:
    # monitor the event log file currently being written by API GW
    # --> REPLACE path to api gateway logs
    - [api gateway install dir]/events/group-1_instance-2*
    # monitor the event log files that the API GW node manager has already processed
    #  Used if API Analytics is running simultaneously with Embedded Analytics
    # --> REPLACE path to api gateway logs or remove it if there is no API Analytics
    - [api gateway install dir]/events/processed/group-1_instance-2_*.log.PROCESSED
  # new files are detected within 10s after they're created
  scan_frequency: 10s
  # new lines in a known file are detected within 1s afer they're written
  backoff: 1s
  # files are closed after 1 minutes without activity
  close_inactive: 1m
  # files older than 24h are ignored
  ignore_older: 24h
  # clean entries files older than 2 days of the registry file
  clean_inactive: 48h
  fields:
    # group matches the group id in the monitored files
    group: 1
    # instance matches the instance id in the monitored files
    instance: 2
    # Back-End since this instance is accessed directly by end users
    type: Back-End


#========================= Filebeat global options ============================
# Name of the registry file. Remembers its current position in each monitored file
# --> REPLACE filebeat registry directory
filebeat.registry_file: [filebeat install dir]/.filebeat


#================================ Outputs ======================================
# Configure Logstash output for Decision Insight to use when sending the data collected
# by the beat.

#----------------------------- Logstash output ---------------------------------
output.logstash:
  # Boolean flag to enable or disable the output module.
  enabled: true

  # The Decision Insight hosts
  # --> REPLACE with the IP address of ADI and Lumberjack port
  hosts: ["adi.collector.node.ip:5044"]

  # Number of workers per Logstash host.
  workers: 4

  # filebeat will not compress data sent to ADI
  compression_level: 0

  # Enable SSL support. SSL is automatically enabled, if any SSL setting is set.
  ssl.enabled: true

  # List of supported/valid TLS versions.
  ssl.supported_protocols: [TLSv1.2]

  # List of root certificates for server verifications
  # --> REPLACE path to root certificate
  ssl.certificate_authorities: ["[filebeat install dir]/myCA.pem"]

  # client certificate + key for client auth (if requested by server)
  # --> REPLACE path to client certificate
  ssl.certificate: "[filebeat install dir]/cert.pem"

  # Client Certificate Key
  # --> REPLACE path to certificate Key
  # key encrypted in aes-256-cbc
  ssl.key: "[filebeat install dir]/certkey.pem"

  # Optional passphrase for decrypting the Certificate Key
  # --> REPLACE the passphrase
  ssl.key_passphrase: "axway*"

  # Configure cipher suites to be used for SSL connections
  ssl.cipher_suites: [ECDHE-RSA-AES-256-GCM-SHA384]


#================================ Logging ======================================

# Sets log level. The default log level is info.
# Available log levels are: critical, error, warning, info, debug
logging.level: warning

# Logging to rotating files files. Set logging.to_files to false to disable logging to
# files.
logging.to_files: true
logging.files:
  # Configure the path where the logs are written. The default is the logs directory
  # under the home path (the binary location).
  # --> REPLACE log directory path
  path: [logging-directory]

  # The name of the files where the logs are written to.
  name: filebeat.log

  # Number of rotated log files to keep. Oldest files will be deleted first.
  keepfiles: 7

File System

If the "[api gateway install dir]/events/processed" folder exist, it must be on the same file system than the "[api gateway install dir]/events" folder to prevent decreases performance.

This configuration line about processed log can be removed if there is no API Analytics.

Each update of the Filebeat configuration file requires a Filebeat service restart.

Custom fields

 For each instance, the custom fields "group (number)" , "instance (number)" and "type (Front-End or Back-End)" should be correctly provisioned.

Resources consumption

In a typical use case the resources footprint of Filebeat is very low.

Nonetheless, the hardware resources consumed by Filebeat can get quite high if Filebeat needs to simultaneously monitor a large number of files.
To avoid any problem:

  • the ignore_older parameter is set by default to 24h
  • the close_inactive parameter is set by default to 1m.
  • the clean_inactive parameter is set by default to 2d.

This allows Filebeat to run without any issue on any hardware setup that API Gateway supports.

If you need to load more than 24 hours of old data, you must adapt the system resources for Filebeat. The following OS limits must be properly configured:

  • number of open files (ulimit -n)
  • number of threads (ulimit -u)
  • virtual memory consumption (ulimit -v)
  • the swap size (grep SwapTotal /proc/meminfo)

workers

For more efficiency, you can grow the number of workers per host. It's depend of throughput of log messages to send to Decision Insight and of your network infrastructure.

8 workers is for high throughput (> 1000 messages per second).

YAML Syntax

Beware, the YAML syntax is very strict. For example, tab characters are not allowed. However, your text editor might automatically use them for indentations without you noticing. Sometimes Filebeat reports the error, sometimes not. Verifying the configuration file with a YAML syntax checker might help.

Related Links