Perform essential Cassandra operations

Tip   A new guide is available in the latest release of API Gateway that includes information on Apache Cassandra best practices and tuning, setting up high availability, and backup and restore. Much of the information in this guide also applies to API Gateway 7.5.3. See the API Gateway 7.6.2 Apache Cassandra Administrator Guide.

This topic describes the minimum essential operations that are required to maintain a healthy Apache Cassandra high availability (HA) cluster.

Repair inconsistent nodes

Apache Cassandra has different ways to maintain data consistency across the cluster during normal runtime processing. However, to resolve any data inconsistencies that could not be resolved by normal runtime processes, you must run an external repair process on a regular basis.

For standard HA configurations, it is best to run repair once per week (during a quiet period) on each node in the cluster.

Caution   The repair must only be executed on one node at a time. You must therefore adjust the repair schedule for each node in the cluster to avoid overlap.

Schedule repair using crontab

When scheduling Cassandra repair events, it is best to use the crontab command of the user running the Cassandra process.

To edit the cron table of this user, execute sudo crontab -u CASSANDRA_USER -e and paste the matching node block from the following examples:

Example cron table entries for three-node cluster repairing one node per day (Saturday to Monday)

Node 1:

Run full repair at 1 a.m. every Saturday:

0 1 * * 6 PATH_TO_CASSANDRA/bin/nodetool CONNECTION_SECURITY_PARAMS repair -pr --full > PATH_TO_CASSANDRA/logs/last_repair.log 2>&1

Node 2:

Run full repair at 1 a.m. every Sunday:

0 1 * * 0 PATH_TO_CASSANDRA/bin/nodetool CONNECTION_SECURITY_PARAMS repair -pr --full > PATH_TO_CASSANDRA/logs/last_repair.log 2>&1

Node 3:

Run full repair at 1 a.m. every Monday:

0 1 * * 1 PATH_TO_CASSANDRA/bin/nodetool CONNECTION_SECURITY_PARAMS repair -pr --full > PATH_TO_CASSANDRA/logs/last_repair.log 2>&1

Note   On Windows, you can use the Windows Task Scheduler instead.

For more details on nodetool repair, see the Manual repair: Anti-entropy repair documentation.

Replace dead nodes

If a node is down for more than 10 days, it should be replaced. For details on replacing dead Cassandra nodes, see the Replacing a dead node or dead seed node documentation.

Back up and restore Cassandra data

To backup and restore Cassandra data (online and HA), follow the instructions in the Backing up and restoring data documentation.

Back up configuration and data for disaster recovery

You must back up the following API Gateway and Cassandra configuration and data for disaster recovery:

  • API Gateway group configuration data
  • Back up the API Gateway group configuration data in apigateway/groups/<group-name>/conf. This contains the API Gateway, API Manager and KPS configuration data. Ensure that you back up these important files.
  • API Gateway KPS data
  • Ensure that you back up KPS data regularly on all nodes using the kpsadmin tool. This backs up API Manager and customer KPS data to simple, portable JSON files. These are also independent of Cassandra and Cassandra versions. For more details, see the API Gateway Key Property Store User Guide.
  • Cassandra configuration
  • Ensure that you back up the CASSANDRA_HOME/conf directory on all nodes.
  • Cassandra data
  • With Cassandra, any node in the system can act as a live backup when replacing a dead node. Data can be restored without affecting availability using Cassandra HA backup and restore. However, it still makes sense to make a regular hard file system backup of the data on all nodes for disaster recovery use cases. Cassandra data resides in CASSANDRA_HOME/data. You should back up this directory on all nodes.
Note   You must stop the Cassandra server before taking a file system backup. This ensures that files are not locked and that all data is flushed to disk. For details, see Install an Apache Cassandra database.

Ensure time is in sync across the Cassandra cluster

You should use a time service such as NTP to ensure that time is in sync in the cluster. For example, OAuth Tokens are removed by Cassandra when expired, based upon the time-to-live value set when created. If Cassandra nodes run on servers that are not synchronized, the tokens are removed at different times, and the data is no longer fully synchronized.

Note   You must ensure that the NTP server is on a physical machine, and not a virtual machine, in the datacenter.

For more details, see https://blog.logentries.com/2014/03/synchronizing-clocks-in-a-cassandra-cluster-pt-2-solutions/

Reconfigure an existing Cassandra installation from scratch

There is no need to reinstall Cassandra from scratch. Instead, you can move Cassandra data files and restore the cassandra.yaml configuration file if necessary (if you updated Cassandra configuration). Perform the following steps:

  1. Stop Cassandra. For details, see Install an Apache Cassandra database.
  2. Move CASSANDRA_HOME/data to CASSANDRA_HOME/data/OLD-DATA-DATE.
  3. Restore cassandra.yaml in CASSANDRA_HOME/conf if necessary.

Enable Cassandra debug logging

You can enable Cassandra debug logging using any of the following approaches:

  • logback.xml
  • You can specify a logger in the cassandra/conf/logback.xml configuration file as follows:

<logger name "org.apache.cassandra.transport" level=DEBUG/>

  • nodetool
  • You can use the nodetool setlogginglevel command as follows:

nodetool setlogginglevel org.apache.cassandra.transport DEBUG

  • JConsole
  • The JConsole tool enables you to configure Cassandra logging using JMX. For example, you can invoke setLoggingLevel and specify org.apache.cassandra.db.StorageServiceMBean and DEBUG as parameters. For more details, see the next section.

For more details on enabling logging, see the following, which also applies to Cassandra 2.2:

Monitor a Cassandra cluster using JMX

You can use Java Management Extensions to monitor and manage performance in a Cassandra cluster. For details, see the Monitoring a Cassandra cluster documentation.

Upgrade your Cassandra version

For details on upgrading your Cassandra version, see Supported Cassandra versions.

Further details

For more details on Cassandra operations, see the Apache Cassandra 2.2 documentation.

Related Links