Manage the system throttle

The Activator system throttle prevents JVM node hard-crashes due to overload. The system throttle does not prevent processing, but rather limits the consumption of new work. The system throttle operates automatically. Throttle behavior is normally applied separately for each node of the cluster. You can also engage the throttle manually on all nodes of the cluster enable the system to finish processing all active messages. The occasional automatic engagement of a system throttle is good for Activator, and is a normal operating experience. When the system throttle engages, it does not mean that the JVM is not working at all, rather, it means that the JVM is working on tasks already in queue, until sufficient headroom is regained to accept new tasks.

The system throttle can be engaged for different reasons:

  • Intentionally engaged at startup (via tuning.properties file setting).
  • Intentionally engaged during runtime (via a UI control button).
  • The trading engine Task Scheduler load exceeds the default limit of 150.
  • The JVM (trading engine or control node) experiences an OutOfMemory exception.
  • The file system takes too long to write a test file to the backup directory. In rare instances, Java Garbage Collection by the control node or trading engine node can require so much time to complete that the timer for the file system health check (when invoked just prior to the onset of a Garbage Collection event) exceeds the 15 second default limit, due to activity suspension until the Garbage Collection completes).

When a system throttle is engaged, consumption of new inbound work is halted. The trading engine stops polling and returns a 503 error to your trading partners when they try to connect.

Activator then re-prioritizes all traffic based upon these rules:

  • New outbound files are consumed, packaged, and sent before any other work.
  • Inbound connections are then opened and inbound messages are processed.
  • Failed outbound/inbound messages are then processed.

When the system throttle is invoked you may receive a log entry similar to the following (TaskScheduler load example):

Message Tracker dependency

When system throttling is engaged, the Message Tracker resend and reprocess attempts fail. Resend and reprocess attempts eventually generate "max attempt exceeded" errors.

System throttle management

Note   System throttle settings should be changed only as part of a comprehensive tuning effort, best conducted with the help of Axway Professional Services.

You manage general system throttle settings from the tuning.properties file. These settings are described in the following paragraphs. Additionally, you can manually engage/disengage system throttling at runtime in the UI.

The tuning.properties file

You manage the general system throttle behavior from the tuning.properties file, located in <install_directory>/conf.

The properties in this file are applied only to the node where the tuning.properties file is located. You must set the property for each node of a cluster by modifying the file for each node.

By default, tuning.properties is empty. This indicates that all of its possible entries are operating at their default values.

This chapter describes how to use the two properties related to system throttling that you can set in this file:

  • systemThrottle.pausePickups – Default=false
  • systemThrottle.maximumTaskQueueSize – Default=150

Force throttling on startup

In some cases, you may want to force system throttling on startup. This is useful, for example, if you want to empty the system of messages waiting to be processed.

To to force startup in throttled mode:

  1. Go to <install_directory>/conf and open the tuning.properties file in a text editor.
  2. Add the property:
  3. systemThrottle.pausePickups
  4. Set the value of the property to "true":
  5. systemThrottle.pausePickups=true
  6. Save the file.
  7. Restart Activator.

When the system restarts, it operates in throttled mode. This status is displayed for trading engine nodes on the System Management page of the UI. You can temporarily override throttling by clicking Restart pickups for all trading engine nodes. When you restart Activator, it restarts in throttled mode until you modify this property.

Modify the Task Scheduler load limit

By default, when the trading engine Task Scheduler load exceeds 150, the system throttle is engaged. You can modify this value. While there is no real limit to this value, there are practical limits due to environmental factors (physical memory size, server hardware, network speed, etc.).

Axway recommends that you change this property value in increments of 25.

To change the Task Scheduler load limit:

  1. Go to <install_directory>/conf and open the tuning.properties file in a text editor.
  2. Add the property:
  3. systemThrottle.maximumTaskQueueSize
  4. Set the value of the property to the desired value, for example:
  5. systemThrottle.maximumTaskQueueSize=175
  6. Save the file.
  7. Restart Activator.

When the system restarts, it engages throttling at the new load limit threshold.

Related Links