Apache Cassandra best practices
Follow the best practices in this section to achieve a stable Apache Cassandra environment, and to prevent data integrity and performance issues.
You must complete all of these tasks before you start Apache Cassandra.
Clock synchronization and health check
The clocks of the system across all Cassandra cluster machines and the clocks of all client machines (API Gateway hosts) must be synchronized to one (1) millisecond precision.
Failing to synchronize the clocks will result in:
- Faults in data synchronization
- Failure to start or configure the machines correctly
The clock synchronization requires the use of a time service, such as NTP (Network Time Protocol), to ensure that the time is synchronized across all machines in the cluster.
You must also perform a health check of the clock drift between nodes on a regular basis.
You must create a specific UNIX user account for the Cassandra database. This user account must own all Cassandra related files, and it must be used to run the Cassandra process.
This guide refers to this user account as
Required system tuning
Cassandra executes basic performance and tuning checks at startup, and it writes warning messages to the console and to the system log file when issues are found.
These are examples of warnings for a misconfigured Cassandra host:
WARN Unable to lock JVM memory (ENOMEM). This can result in part of the JVM being swapped out, especially with mmapped I/O enabled. Increase RLIMIT_MEMLOCK or run Cassandra as root
WARN jemalloc shared library could not be preloaded to speed up memory allocations
WARN Cassandra server running in degraded mode. Is swap disabled? : true, Address space adequate? : true, nofile limit adequate? : false, nproc limit adequate? : false
Perform the following procedures to ensure that each Cassandra machine meets the basic tuning requirements.
||The following commands apply for Red Hat 7.x. If you are using another UNIX/Linux distribution, consult your system administrator.
jemalloc is installed.
- Run the command
rpm -q jemalloc to check if jemalloc is installed.
- If jemalloc is not installed, run the following command to install it from
sudo rpm -iv https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm
sudo yum install -y jemalloc
Turn swap off
Ensure that you turn
||Apache Cassandra recommends that
swap is disabled. If your company policies or production environment requires
swap to be
on for other processes, you must ensure that the Cassandra process is not
swapped out at any time.
- The line
cat/proc/swaps in the
/etc/fstab file should show
NO entries. If entries are present, execute the following command to disable all swap entries currently active:
- Delete all swap entries in
/etc/fstab to ensure that swap is not enabled again when the machine is restarted.
Set limits for user account
Set the minimum limits for the user account.
If running via console or
init.d, create a
/etc/security/limits.d/cassandra.conf file, and add the following lines to it (replace
cassandra_user with the relevant user account).
<cassandra_user> - memlock unlimited
<cassandra_user> - nofile 100000
<cassandra_user> - nproc 32768
<cassandra_user> - as unlimited
If running via a system service, ensure that the following lines are present in the
[SERVICE] section of the Cassandra service file:
Clean up Cassandra repair history
By default Apache Cassandra 2.2.x does not clean up
nodetool repair trace history. This can cause the
system_distributed keyspace to increase in size over time. The extent of the issue can be seen by running the following command to see how much space is being consumed by
du -md 1 <cassandra_root>/data/data/ | sort -n
To prevent this, it is recommended that you set a 7 day TTL on the repair history tables and remove any existing data. First, execute the following using
cqlsh on one of the Cassandra nodes:
ALTER TABLE system_distributed.repair_history WITH default_time_to_live = 604800;
ALTER TABLE system_distributed.parent_repair_history WITH default_time_to_live = 604800;
To reclaim the disk space, clean up the snapshots generated by the truncate by executing the following against all Cassandra nodes:
nodetool clearsnapshot system_distributed