Filter transactions in Prebuilt Dashboards

Introduction

If you aim to use the Prebuilt Dashboards to analyze  only a subset of the events that API Gateways records, filtering transactions is useful.

Filtering transactions has many benefits:

  • Avoids pollution of the Prebuilt Dashboards with non required data.
  • Allows better general performances.
  • Saves resources (storage, CPU, RAM, network bandwidth).

Filtering transactions can be configured at Filebeat level or in an Axway Decision Insight (DI) route.

Filter messages with Filebeat

With a specific configuration, Filebeat will not forward to DI all the events that API Gateway records.

Pros and cons of filtering with Filebeat:

(plus) Filtering is done at the lowest level thus saves more resources on the DI node.

(plus) Saves network bandwidth between the API Gateway node and the DI node.

(minus) Less filtering options than using routes: the filtering is based on regular expressions.

(minus) Filtering can raise CPU usage on the API Gateway node - this should be negligible in most cases.

(minus) Deployment and maintenance are slightly more cumbersome:

  • a Filebeat configuration update is needed for each API gateway node,
  • Filebeat needs a restart after a configuration update,
  • Regular expressions can become hard to maintain.

Implementation

In the filebeat.yml configuration file, each prospector can be configured with the include_lines and exclude_lines parameters. These parameters rely on regular expressions.

The complete Filebeat documentation on these parameters is available here and the supported regular expressions syntax is presented here.

Examples

  • In the following configuration, Filebeat does not forward transaction events where the serviceName field is null:
filebeat.yml
filebeat:
  prospectors:
    - paths:
        - /opt/Axway/API/apigateway/events/group-1_instance-1.log 
        - /opt/Axway/API/apigateway/events/processed/group-1_instance-1_*.log.PROCESSED
      exclude_lines: [ '"serviceName":null' ] 
      scan_frequency: 10s
      backoff: 1s
      close_inactive: 1h
      ignore_older: 24h
      clean_inactive: 48h
      fields:
        group: 1
        instance: 1
        type: Front-End
  registry_file: /opt/Axway/filebeat/.filebeat
output:
  logstash:
    hosts: ["adi.collector.node.ip:5044"]
    compression_level: 0
    ssl:
      certificate_authorities: ["/opt/Axway/filebeat/myCA.pem"]
      certificate: "/opt/Axway/filebeat/cert.pem"
      key: "/opt/Axway/filebeat/certkey.pem"
      supported_protocols: [TLSv1.2]
      cipher_suites: [ECDHE-RSA-AES-256-GCM-SHA384]
  • In the following configuration, Filebeat only forwards events of type "header", "system" or "transaction":
filebeat.yml
filebeat:
  prospectors:
    - paths:
        - /opt/Axway/API/apigateway/events/group-1_instance-1.log 
        - /opt/Axway/API/apigateway/events/processed/group-1_instance-1_*.log.PROCESSED
      include_lines: [ '^{"type":"header"', '^{"type":"system"', '^{"type":"transaction"' ] 
      scan_frequency: 10s
      backoff: 1s
      close_inactive: 1h
      ignore_older: 24h
      clean_inactive: 48h
      fields:
        group: 1
        instance: 1
        type: Front-End
  registry_file: /opt/Axway/filebeat/.filebeat
output:
  logstash:
    hosts: ["adi.collector.node.ip:5044"]
    compression_level: 0
    ssl:
      certificate_authorities: ["/opt/Axway/filebeat/myCA.pem"]
      certificate: "/opt/Axway/filebeat/cert.pem"
      key: "/opt/Axway/filebeat/certkey.pem"
      supported_protocols: [TLSv1.2]
      cipher_suites: [ECDHE-RSA-AES-256-GCM-SHA384]

Filter messages in data integration routes

Implementation

You can implement the events filtering by updating the already existing route below, located around line 30 of the routing context collectAndProcessEvents in space 04-API-Integration.

  <route><from uri="direct:api_gw_filter_event"/>
    <filter>
       <simple>true</simple>     
       <to uri="direct:api_gw_process_event"/>     
    </filter>
  </route>

As an example, this filtering route can be updated as follows, to exclude events of type 'remoteHostDown' and 'remoteHostUp':

  <route><from uri="direct:api_gw_filter_event"/>
    <!-- placeholder for event filters -->
    <filter>
       <simple>${body[type]} != 'remoteHostDown' &amp;&amp;  ${body[type]} != 'remoteHostUp'</simple>     
       <to uri="direct:api_gw_process_event"/>     
    </filter>
  </route>

Related Links