Global caches

In cases where a back-end service is serving the same request (and generating the same response) over and over again, it makes sense to use a caching mechanism. When a cache is employed, a unique identifier for the request is cached together with the corresponding response for this request. If an identical request is received, the response can be retrieved from the cache instead of forcing the service to reprocess the identical request and generate the same response. The use of caching in this way helps divert unnecessary traffic from the service and makes it more responsive to new requests.

For example, assume you have deployed a service that returns a list of cities in the USA from an external database, which is then used by a variety of web-based applications. Because the names and quantity of cities in the USA are relatively constant, if the service handles hundreds or thousands of requests every day, this is waste of processing time and effort, especially considering that the database that contains the relatively fixed list of city names is hosted on a separate machine to the service.

If you assume that the list of cities in the database does not change very often, it makes sense to use API Gateway to cache the response from the service that contains the list of cities. When API Gateway identifies a request for this service, the cached response can be returned to the client. This approach results in the following performance improvements:

  • API Gateway does not have to route the message on to the service, therefore saving the processing effort required, and perhaps more importantly, saving the time it takes for the round trip.
  • The service does not have to waste processing power on generating the same list over and over again, therefore making it more responsive to requests for other services.
  • Assuming a naive implementation of database retrieval and caching, the service does not have to query the database (over the network) and collate the results over and over again for every request.

The caching mechanism used in API Gateway offers full control over the size of the cache, the lifetime of objects in the cache, whether objects are cached to disk, and even whether caches can be replicated across multiple API Gateway instances. This topic describes how to configure both local and distributed caches in API Gateway, and shows examples of how to configure a policy to cache responses.

Local caches

Local caches are used where a single API Gateway instance has been deployed. In such cases, you do not need to replicate caches across multiple running API Gateway instances.

Add a local cache

  1. In the Policy Studio node tree, click Environment Configuration > Libraries > Caches, and clicking the Add button at the bottom right of the window.
  2. Select Add Local Cache.
  3. Configure the following fields on the Configure Local Cache dialog:
Field Description
Cache Name Enter a name for the cache.
Maximum Elements in Memory Enter the maximum number of objects that can be in memory at any one time.
Maximum Elements on Disk Set the maximum number of objects that can be stored in the disk store at any one time. A value of zero indicates an unlimited number of objects.
Eternal Select this option if you want that objects stored in the caches never expire and timeouts have no effect.
Overflow to Disk Select this option if you want the cache to overflow to disk when the number of objects in memory has reached the amount set in the Maximum Elements in Memory field above.
Time to Idle

Set the maximum amount of time (in seconds) between accesses that an object can remain idle before it expires. A value of zero indicates that objects can idle for infinity, which is the default value. If you select Eternal, this setting is ignored.

Time to Live Set the maximum time between when an object is created and when it expires. The default value is zero, which means that the object can live for infinity. If you select Eternal, this setting is ignored.
Persist to Disk Select this option if you want that the disk store is persisted between JVM restarts. This option is disabled by default.
Disk Expiry Interval Set the number of seconds between runs of the disk expiry thread. The default is 120 seconds.
Disk Spool Buffer Size Set the size of memory (in MBs) to allocate the disk store for a spool buffer. Writes are made to this memory and then asynchronously written to disk. The default size is 30 MB. If you get OutOfMemory exceptions, you can consider lowering this value. However, if you notice poor performance, increase the value.
Eviction Policy Select the eviction policy that the cache uses to evict objects from the cache. The default policy is Least Recently Used. However, you can also use First in First Out and Less Frequently Used.

Distributed caches

If you have deployed several API Gateways throughout your network, you may need to employ a distributed cache. In this scenario, each API Gateway has its own local copy of the cache but registers a cache event listener that replicates messages to the other caches so that put, remove, expiry, and delete events on a single cache are duplicated across all other caches.

Add a distributed cache

  1. In the Policy Studio node tree, click Environment Configuration > Libraries > Caches, and clicking the Add button at the bottom right of the window.
  2. Select Add Distributed Cache.
  3. Configure the fields on the Configure Distributed Cache dialog:
  • API Gateway only supports the RMICacheReplicatorFactory in Ehcache. Other replication factories are not supported.
  • Many of the settings for the distributed cache are identical to those for the local cache. For details on how to configure these fields, see Add a local cache. The following table lists the fields only displayed in the distributed cache dialog.
Field Description
Event Listener Class Name Enter the name of the listener factory class that enables this cache to register listeners for cache events, such as put, remove, delete, and expire.
Properties Separator Specify the character to use to separate the list of properties.

Specify the properties to pass to the RMICacheReplicatorFactory. The following properties are available:

  • replicatePuts=true | false: Determines whether new elements placed in a cache are replicated to other caches. Default is true.
  • replicateUpdates=true | false: Determines whether new elements that override (update) existing elements with the same key in a cache are replicated. Default is true.
  • replicateRemovals=true: Determines whether element removals are replicated. Default is true.
  • replicateAsynchronously=true | false: Determines whether replications are asynchronous (true) or synchronous (false). Default is false.
  • replicateUpdatesViaCopy=true | false: Determines whether new elements are copied to other caches (true)or a remove message is sent (false). Default is true.
  • asynchronousReplicationIntervalMillis=[number of ms]: Sets the interval (in milliseconds) the asynchronous replicator runs at. The default is 1000 and the minimum is 10. This property is only applicable if replicateAsynchronously=true.
Cache Bootstrap Class Name Specify a BootstrapCacheLoader factory that the cache can call on initialization to prepopulate itself. The RMIBootstrapCacheLoader bootstraps caches in clusters where RMICacheReplicators are used.
Properties Separator Specify the character to use to separate the list of properties.

List the properties that are used to initialize the RMIBootstrapCacheLoaderFactory. The following properties are recognized:

  • bootstrapAsynchronously=true | false: Determines whether the bootstrap happens in the background after the cache has started (true), or if bootstrapping must complete before the cache is made available (false). Default is true.
  • maximumChunkSizeBytes=[integer]: Caches can potentially grow larger than the memory limits on the JVM. This property enables the bootstrapper to fetch elements in chunks. The default chunk size is 5000000 (5 MB).

Global distributed cache settings

In a distributed cache, there is no master cache controlling all caches in the group. Instead, each cache is a peer in the group and needs to know where all the other peers in the group are located. Peer Discovery and Peer Listeners are two essential parts of any distributed cache system.

Edit global distributed cache settings for peer discovery

  1. In the Policy Studio node tree, click Server Settings > General > Cache.
  2. Configure the following fields:
Field Description
Peer Provider Class By default, the built-in peer discovery class factory is used: net.sf.ehcache.distribution.RMICacheManagerPeerProviderFactory
Properties Separator Specify the token used as the separator for the list of properties in the next field.

The properties listed specify whether the peer discovery mechanism is automatic or manual. If the automatic mechanism is used, each peer uses TCP multicast to establish and maintain a multicast group. This is the default option because it requires minimal configuration and peers can be automatically added and removed from the group. Each peer pings the group every second. If a peer has not pinged any of the other peers after 5 seconds, it is dropped from the group, while a new peer is admitted to the group if it starts pinging the other peers.

To use automatic peer discovery, ensure that the peerDiscovery setting is set to automatic. You can specify the multicast address and port using the multicastGroupAddress and multicastGroupPort settings. You can specify the time to live for multicast datagrams using the timeToLive setting.

Alternatively, you can configure a manual peer discovery mechanism, whereby each peer definitively lists the peers it wants to communicate with. This should only be used in networks where there are problems propagating multicast datagrams. To use a manual peer discovery mechanism, ensure the peerDiscovery setting is set to manual. The list of RMI URLs of the other peers in the group must also be specified, for example: rmiUrls=//server2:40001/sampleCache1|//server2:40001/sampleCache2

Peer Listener Class Specify the peer listener class responsible for listening for messages from peers in the group.
Properties Separator Specify the token used to separate the list of properties.
Properties The properties entered configure the way the listener behaves. Valid properties are as follows:
  • hostname (optional): Host name of the machine on which the listener is listening.
  • By default, the host name is set to localhost, which maps to the local loopback address of This hostname is not addressable from another machine on the network. If you intend this cache to be used over the network, you must change this address to the IP address of the network interface on which the listener is listening.
  • port (mandatory): Specify the port on which the listener is listening. The default port is 40001.
  • socketTimeoutMillis (optional): Enter the number of seconds that client sockets wait when sending messages to this listener until they give up. The default is 2000 ms.
Notify replicators of removal of items during refresh

A server refresh automatically purges all items from the cache (for example, when configuration updates are deployed to API Gateway). If this check box is selected, the contents of each peer in the group are also purged. This avoids a situation where a single peer is refreshed (and has its contents purged), but the other peers in the group are not purged.

If this option is not selected, the refreshed peer attempts to bootstrap itself to the other peers in the group, resulting in the cache items becoming replicated in the refreshed cache. This effectively negates the effect of the server refresh and may result in inconsistent behavior.

Distributed caching example

This example describes how to configure Ehcache for three API Gateways:

  • You have three API Gateway servers and two distributed caches.
  • Each API Gateway server shares its cache with the other two servers.

Step 1 – Change Ehcache settings in Policy Studio

In Policy Studio, select the Server Settings node in the Policy Studio tree, and click General > Cache.

Edit the peer provider properties as follows:


Edit the peer listener properties as follows:


Step 2 – Configure envSettings.props for each API Gateway server

Add the following settings to the envSettings.props file for each API Gateway server.

envSettings.props for API Gateway 1


envSettings.props for API Gateway 2


envSettings.props for API Gateway 3


Example of caching response messages

This simple example shows how to construct a policy that caches responses from the service. It uses the request body to identify identical successive requests. If API Gateway receives two successive requests with an identical message body, it returns the corresponding response from the cache instead of routing the request to the service.

The following diagram illustrates the complete policy:

Caching Response Messages

The logic of the policy is summarized as follows:

  1. The Create Key filter defines what part of the request is used as unique key to identify unique requests. The key is then used to look up the appropriate response message from the cache.
  2. The Is Cached? filter looks up the request body in the response cache to see if the cache already contains the request body. If it does, the Reflect filter returns the correspoding response message to the client.
  3. If corresponding request body is not found, the Routing filter routes the request to the service to be processed. The service then returns a response to API Gateway.
  4. API Gateway returns the response to the client, and the Cache attribute filter caches the response.
  5. When API Gateway receives the next identical request, the corresponding response is located in the responses cache and returned immediately to the client.

See the following example configurations of the caching filters to achieve this policy. The routing filters are not included in this example because the configuration options depend on your target service.

Create Key

This filter defines what part of the request you want to use to identify unique requests. Different parts of the request can be identified internally using message attributes (for example, content.body contains the request body). You must configure the following fields:

  • Name: Use request body to create unique key
  • Attribute Name: content.body
  • Output attribute name: message.key

Is Cached?

This filter searches the cache to see if a response has been stored for the current request. By default, the filter uses the message.key attribute to search the cache. The message.key attribute contains a hash of the request message, and can be used as the key for objects in the cache. You must configure the following fields:

  • Name: Is a response for this request already cached?
  • Cache containing key: Response Cache (assuming you have created a cache of this name)
  • Attribute Containing Key: message.key
  • Overwrite attribute name if found: content.body

If the key is found in the cache, the value of the key (cached response for this request) is written to the content.body attribute, which can be returned to the client using the Reflect filter.


If the Is Cached? filter passes, it retrieves the response from the cache and stores it in the content.body message attribute. The Reflect filter returns the cached response to the client.


If a response for this request was not found in the cache, API Gateway routes the request to the service, and waits for a response. For more details on how to route messages, see Get started with routing configuration.

Cache Attribute

When the response is received from the service, it should be cached for future use. The Cache Attribute filter defines the key that is used to search the cache as well as which aspect of the response message is stored as the key value in the cache.

Note   This example specifies the value of the content.body attribute to cache. Because this filter is configured after the routing filters, this attribute contains the response message. The value entered in the Attribute Key field should match that entered in the Attribute containing key field in the Is Cached? filter.

You must configure the following fields:

  • Name: Cache response body
  • Cache to use: Response Cache
  • Attribute key: message.key
  • Attribute name to store: content.body

Further information

For more information on these filters, see the API Gateway Policy Developer Filter Reference.

Related Links