Apache Cassandra best practices

Follow the best practices in this section to achieve a stable Apache Cassandra environment, and to prevent data integrity and performance issues. You must complete all of these tasks before you start Apache Cassandra.

Clock synchronization and health check

The clocks of the system across all Cassandra cluster machines and the clocks of all client machines (API Gateway hosts) must be synchronized to one (1) millisecond precision.

Failing to synchronize the clocks will result in:

  • Faults in data synchronization
  • Failure to start or configure the machines correctly

The clock synchronization requires the use of a time service, such as NTP (Network Time Protocol), to ensure that the time is synchronized across all machines in the cluster.

You must also perform a health check of the clock drift between nodes on a regular basis.

User account

You must create a specific UNIX user account for the Cassandra database. This user account must own all Cassandra related files, and it must be used to run the Cassandra process.

This guide refers to this user account as cassandra_user.

Required system tuning

Cassandra executes basic performance and tuning checks at startup, and it writes warning messages to the console and to the system log file when issues are found.

These are examples of warnings for a misconfigured Cassandra host:

WARN Unable to lock JVM memory (ENOMEM). This can result in part of the JVM being swapped out, especially with mmapped I/O enabled. Increase RLIMIT_MEMLOCK or run Cassandra as root
WARN jemalloc shared library could not be preloaded to speed up memory allocations
WARN Cassandra server running in degraded mode. Is swap disabled? : true, Address space adequate? : true, nofile limit adequate? : false, nproc limit adequate? : false

Perform the following procedures to ensure that each Cassandra machine meets the basic tuning requirements.

Note   The following commands apply for Red Hat 7.x. If you are using another UNIX/Linux distribution, consult your system administrator.

Install jemalloc

Ensure that jemalloc is installed.

  1. Run the command rpm -q jemalloc to check if jemalloc is installed.
  2. If jemalloc is not installed, run the following command to install it from epel:
sudo rpm -iv https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm
sudo yum install -y jemalloc

Turn swap off

Ensure that you turn swap off.

Note   Apache Cassandra recommends that swap is disabled. If your company policies or production environment requires swap to be on for other processes, you must ensure that the Cassandra process is not swapped out at any time.
  1. The line cat/proc/swaps in the /etc/fstab file should show NO entries. If entries are present, execute the following command to disable all swap entries currently active:
  2. sudo swapoff -a sad
  3. Delete all swap entries in /etc/fstab to ensure that swap is not enabled again when the machine is restarted.

Set limits for user account

Set the minimum limits for the user account.

If running via console or init.d, create a /etc/security/limits.d/cassandra.conf file, and add the following lines to it (replace cassandra_user with the relevant user account).

<cassandra_user> - memlock unlimited
<cassandra_user> - nofile 100000
<cassandra_user> - nproc 32768
<cassandra_user> - as unlimited

If running via a system service, ensure that the following lines are present in the [SERVICE] section of the Cassandra service file:

LimitMEMLOCK=infinity
LimitNOFILE=100000
LimitNPROC=32768
LimitAS=infinityd

Clean up Cassandra repair history

By default Apache Cassandra 2.2.x does not clean up nodetool repair trace history. This can cause the system_distributed keyspace to increase in size over time. The extent of the issue can be seen by running the following command to see how much space is being consumed by system_distributed:

du -md 1 <cassandra_root>/data/data/ | sort -n

To prevent this, it is recommended that you set a 7 day TTL on the repair history tables and remove any existing data. First, execute the following using cqlsh on one of the Cassandra nodes:

ALTER TABLE system_distributed.repair_history WITH default_time_to_live = 604800;
TRUNCATE system_distributed.repair_history;
ALTER TABLE system_distributed.parent_repair_history WITH default_time_to_live = 604800;
TRUNCATE system_distributed.parent_repair_history;

To reclaim the disk space, clean up the snapshots generated by the truncate by executing the following against all Cassandra nodes:

nodetool clearsnapshot system_distributed

Related Links