Configure a highly available Cassandra cluster

This topic describes how to set up an Apache Cassandra database cluster for high availability (HA) of your API Gateway system.

Cassandra HA in a production environment

To tolerate the loss of one Cassandra node and to ensure 100% data consistency, API Gateway requires at a minimum the following Cassandra cluster configuration running in a production class environment:

  • Three Cassandra nodes (with one seed node)
  • QUORUM read/write consistency to ensure that you are reading from a quorum of Cassandra nodes (two) every time
  • Caution   Eventual consistency is not supported in a production environment due to a risk of stale or missing data
  • Replication factor setting set to 3, so each node holds 100% of the data

If one Cassandra node fails or needs to be restarted, the cluster continues to operate with the remaining two nodes. This configuration applies to all supported use cases (for example, API Manager and API Gateway custom KPS, OAuth, and client registry data).

Preparing a Cassandra HA environment

The following general guidelines apply to configuring a Cassandra HA cluster:

  • Decide on the number of Cassandra nodes, and install and configure the Cassandra database on each node.
  • Decide on the number of API Gateway nodes, and install and configure them (local or remote to Cassandra).
  • Note   It is recommended that you configure a Cassandra HA cluster with three Cassandra nodes, and at least two API Gateway instances (local or remote). For details, see Cassandra deployment architectures.
  • Create a HA API Gateway environment.
  • Configure the Cassandra client settings in Policy Studio for the API Gateway group.

For details on how to install an Apache Cassandra database or how to create a HA API Gateway environment, see API Gateway Installation Guide.

Example of a HA setup using three Cassandra nodes

This section shows how to configure a three nodes Cassandra HA cluster.

Configure the Cassandra seed node

Perform the following steps to configure a seed node:

  1. Assign one of the nodes as the seed node, and connect to it via SSH.
  2. Update the CASSANDRA_HOME/conf/cassandra.yaml file manually or using the setup-cassandra script.
    Manually:
  3. seed_provider, parameters, seeds: IP_SEED_NODE
    start_native_transport: true
    start_rpc: false
    native_transport_port: 9042
    listen_address: IP_SEED_NODE
    rpc_address: IP_SEED_NODE
    authenticator: org.apache.cassandra.auth.PasswordAuthenticator
    authorizer: org.apache.cassandra.auth.CassandraAuthorizer
  4. Using the setup-cassandra script:
  5. setup-cassandra --seed --own-ip=<IP_SEED_NODE> --nodes=3 --cassandra-config=CONFIG_FILE

  6. Start Cassandra.
  7. Verify the CASSANDRA_HOME/logs/system.log does not contain any errors or warnings.
  8. Run the nodetool status command to verify that one node is present in UN status, and that the IP address is correct.

Configure the additional Cassandra nodes

The procedure to configure additional nodes uses the seed node that you have previously configured.

Note   You must repeat these steps for each additional node.
  1. Connect to the additional node via SSH, and update the CASSANDRA_HOME/conf/cassandra.yaml file.
    Manually:
  2. seed_provider, parameters, seeds: IP_SEED_NODE
    start_native_transport: true
    start_rpc: false
    native_transport_port: 9042
    listen_address: IP_NODE_N
    rpc_address: IP_NODE_N
    authenticator: org.apache.cassandra.auth.PasswordAuthenticator
    authorizer: org.apache.cassandra.auth.CassandraAuthorizer
  3. Using the setup-cassandra script:
  4. setup-cassandra --seed-ip=<IP_SEED_NODE> --own-ip=<IP_NODE_n> --cassandra-config=<PATH_TO_CASSANDRA.YAML>

  5. Start Cassandra.
  6. Verify the CASSANDRA_HOME/logs/system.log does not contain any errors or warnings.
  7. Run the nodetool status command to verify that the new node is present in UN status, and that the IP address is correct.

Create a new Cassandra database user

From the command line, execute the following commands to create a new user:

>./cqlsh <IP_NODE_1> -u cassandra -p cassandra

> ALTER KEYSPACE "system_auth" WITH REPLICATION = { 'class': 'SimpleStrategy', 'replication_factor': 3 };

> CREATE USER <USERNAME> WITH PASSWORD '<PASSWORD>' SUPERUSER;

> QUIT;

>./cqlsh <IP_NODE_1> -u <ADMIN_USERNAME> -p <PASSWORD>

> ALTER USER cassandra WITH PASSWORD '<PASSWORD>' NOSUPERUSER;

> QUIT

Configure the API Gateway Cassandra client connection

Note   This section assumes that you have already set up a HA API Gateway environment. For details, see API Gateway Administrator Guide

In Policy Studio, open your API Gateway group configuration.

  1. Select Server Settings > Cassandra > Authentication, and enter the cassandra database user name and password.
  2. Select Server Settings > Cassandra > Hosts, and add a host for each Cassandra node in the cluster.
  3. Select Server Settings > Cassandra > Keyspace, and set the Initial replication factor option to 3.
  4. Deploy the configuration to the group.

Configure the client settings for API Gateway or API Manager

Note   You need at least two API Gateways in a group for HA.

This section describes the following options:

Configure API Gateway Cassandra client settings

To update the Cassandra client configuration for API Gateway, perform the following steps:

Configure the API Gateway domain

  1. Ensure API Gateway has been installed on the API Gateway 1 and API Gateway 2 nodes. For details, see the API Gateway Installation Guide.
  2. Ensure an API Gateway domain has been created on the API Gateway 1 node using managedomain. For more details, see Configure an API Gateway domain in the API Gateway Administrator Guide.

Configure the API Gateway Cassandra client connection

  1. In Policy Studio, open your API Gateway group configuration.
  2. Select Server Settings > Cassandra > Authentication, and enter your Cassandra user name and password (both default to cassandra).
  3. Select Server Settings > Cassandra > Hosts, and add an address for each Cassandra node in the cluster (ipA, ipB and ipC in this example).
Tip   You can automate these steps by running the updateCassandraSettings.py script against a deployment package (.fed). For more details, see Configure a highly available Cassandra cluster.

Configure the API Gateway Cassandra consistency levels

  1. Ensure that the API Server KPS collection has been created under Environment Configuration > Key Property Stores. This is required to configure Cassandra consistency levels, and is created automatically if you installed the Complete setup type.
  2.  If you installed the Custom or Standard setup, you must configure OAuth or API Manager settings in Policy Studio to create the required KPS collections. For more details, see:
  3. Select Environment Configuration > Key Property Stores > API Server > Data Sources > Cassandra Storage, and click Edit.
  4. In the Read Consistency Level and Write Consistency Level fields, select QUORUM:
  1. Repeat this step for each KPS collection using Cassandra (for example, Key Property Stores > OAuth, or API Portal for API Manager). This also applies to any custom KPS collections that you have created.
  2. If you are using OAuth and Cassandra, you must also configure quorum consistency for all OAuth2 stores under Libraries > OAuth2 Stores:
    • Access Token Stores > OAuth Access Token Store
    • Authorization Code Stores > Authz Code Store
    • Client Access Token Stores > OAuth Client Access Token Store
Note   By default, OAuth uses EhCache instead of Cassandra. For more details on OAuth, see the API Gateway OAuth User Guide.

Deploy the configuration

  1. Click Deploy in the toolbar to deploy this configuration to the API Gateway group.
  2. Restart each API Gateway in the group.

For details on any connection errors between API Gateway and Cassandra, see Configure a highly available Cassandra cluster.

Configure API Manager Cassandra client settings

To update the Cassandra client configuration for API Manager, perform the following steps:

  1. Ensure the API Gateway and API Manager components have been installed on the API Gateway 1 and API Gateway 2 nodes. These can be local or remote to Cassandra installations. For details, see the API Gateway Installation Guide.
  2. Ensure an API Gateway domain, group, and instance have been created on the API Gateway 1 node using managedomain. For more details, see Configure an API Gateway domain in the API Gateway Administrator Guide.
  3. Note   This section assumes that you have already configured API Manager on the first node in non-HA standalone mode (for example, using Policy Studio or setup-apimanager). For more details, see the API Manager User Guide.
  1. Start the first API Gateway instance in the group. For example:
  2. startinstance -n "my_gw_server_1" -g "my_group"

  3. Configure the Cassandra connection on the API Gateway 1 node. For details, see Configure the API Gateway Cassandra client connection.
  1. Configure the Cassandra consistency levels for your KPS Collections. For details, see Configure the API Gateway Cassandra consistency levels.
  2. In the Policy Studio tree, select Server Settings > API Manager > Quota Settings, and ensure that Use Cassandra is selected.
  3. Under Cassandra consistency levels, in both the Read and Write fields, select QUORUM:
  4. Three node HA full consistency
  1. Add the API Gateway 2 host machine to the domain using managedomain.
  2. Create the second API Gateway instance in the same group on the API Gateway 2 node.
Note   Do not start this instance, and do not configure API Manager for this instance in Policy Studio.
  1. Before starting the second API Manager-enabled instance, ensure that each instance has unique ports in the envSettings.props file. For example:
    1. Edit the envSettings.props file for the API Gateway instance in the following directory:
    1. Add the API Manager ports. For example, the defaults are:
  1. Start the second API Gateway instance. For example:
  2. startinstance -n "my_gw_server_2" -g "my_group"

  3. On startup, this instance receives the API Manager configuration for the group. It now shares the same KPS and Cassandra configuration and data, and uses the ports specified in the envSettings.props file.

Related Links