Apache Cassandra back up and restore

In an Apache Cassandra database the data is replicated among multiple nodes across multiple data centers. Apache Cassandra is highly fault-tolerant, depending on the size of the cluster it can survive the failure of one or more nodes without any interruption in service. However, backups are still needed to recover from the following scenarios:

  • Any errors made in data by client applications
  • Accidental deletions
  • Catastrophic failure that will require the rebuild of the entire cluster
  • In the event of data corruption, when a roll back of the cluster to a known good state is needed

When using an Apache Cassandra cluster where the replication factor is the same as the cluster size, that is each node contains 100% of the data, it's enough to run the data backup procedures only on one of the nodes in the cluster, preferable on the  seed node.

Snapshot based back up

Apache Cassandra does not automatically take backups, but it allows you to create a backup of a node by running the nodetool snapshot command. This creates a hard link for the stable files in the data directory to a schema table.

Running the nodetool snapshot command on a node creates a new directory (named with the Unix timestamp by default) under the snapshots directory.

The following is a sample of a Cassandra data directory structure:

/opt/cassandra/

├── data <- CASSANDRA_DATA_DIRECTORY

│ ├── commitlog <- COMMIT_LOG

│ ├── data

│ │ ├── axwayapimanagement <- KEYSPACE

│ │ │ ├── api_portal_apiapppolicybindings-dca38a70755711e893954770985d2dcd <- TABLE

│ │ │ │ └── snapshots

│ │ │ │ │ └── 1530626296460 <- CURRENT_SNAPSHOT

│ │ │ ├── api_portal_apiorgpolicybindings-de937390755711e893954770985d2dcd <- TABLE

│ │ │ │ └── snapshots

│ │ │ │ │ └── 1530626296460 <- CURRENT_SNAPSHOT

│ │ │ ├── ...

We recommend to take a snapshot on a daily basis.

Note   All the backup files created during snapshot backups are not managed by Cassandra, you must clean them up manually.

Create a script to take the snapshot backup

We recommend that you use a script to copy all CURRENT_SNAPSHOT directories to an external location, making sure that the KEYSPACE and TABLE structures are kept intact, and that snapshots from different nodes are stored separately.

The following is an example of a script to back up the Apache Cassandra data.

#!/bin/bash

#######################################################################

# Sample Cassandra backup script # # NOTE: This MUST be adapted for and validated in your environment before use!

# ######################################################################

set -e

trap '[ "$?" -eq 0 ] || echo \*\*\* FATAL ERROR \*\*\*' EXIT $?

CASSANDRA_DATA_DIR=/opt/cassandra/data

BACKUP_DIR=/backup/cassandra/node_1

BACKUP_TIME=1530626296460

for line in $(find "${CASSANDRA_DATA_DIR}" -type d -name "${BACKUP_TIME}"); do

  keyspace_name="$(basename "$(dirname "$(dirname "$(dirname "${line}")")")")"

  table_name="$(basename "$(dirname "$(dirname "${line}")")")"

  current_backup_dir=${BACKUP_DIR}/${BACKUP_TIME}/${keyspace_name}/${table_name}

  mkdir -p "${current_backup_dir}"

  cp -a "${line}"/* "${current_backup_dir}"

done

Once the script is finished, it creates the following backup directory structure.

/backup/cassandra/node_1

├── 1530626296460 <- SNAPSHOT_DATE

│ ├── axwayapimanagement <- KEYSPACE

│ │ ├── api_portal_apiapppolicybindings-dca38a70755711e893954770985d2dcd <- TABLE

│ │ ├── <SNAPSHOT FILES>

│ │ ├── api_portal_apiorgpolicybindings-de937390755711e893954770985d2dcd <- TABLE

│ │ ├── <SNAPSHOT FILES>

│ │ ├── ...

Create a snapshot

To create a snapshot backup, follow these steps:

  1. Run the nodetool snapshot command on the seed node.
  2. The snapshot files will be created for each active table. (see CURRENT_SNAPSHOT in the sample Cassandra data directory structure).

  3. Run the snapshot backup script.
  4. Run the nodetool clearsnapshot <CURRENT_SNAPSHOT> command to clear the snapshot from the data file directories.

The result of this procedure is a complete backup of all keyspaces and tables (system, system_auth, system_distributed, system_traces, and any <custom keyspaces>).

Restore a snapshot

Restore snapshot to an existing cluster

  1. Shutdown all nodes in the cluster.
  2. Clean up the data directory in each node of the cluster.
    • Delete all *.db files in all TABLE directories.
    • Delete all files in the COMMIT_LOG directory.
  3. Copy the relevant KEYSPACE and TABLE directories from the backup location to the CASSANDRA_DATA_DIRECTORY on the seed node.
  4. Start all nodes in the cluster.
  5. Run a full cluster repair. For details, see Perform anti-entropy repair section in the Perform essential Apache Cassandra operations.

Restore a complete snapshot backup to a new cluster

Note   This operation requires a complete backup of all keyspaces and tables.

To restore a complete snapshot backup to a new cluster, follow the instructions in the Configure a highly available Cassandra cluster, but before starting the seed node, perform the following steps:

  1. Create the CASSANDRA_DATA_DIRECTORY.
  2. Copy all KEYSPACE and TABLE directories from the backup location to the CASSANDRA_DATA_DIRECTORY.

This will load the seed node with the backup data together with the Cassandra database user.

Restore a table from snapshot

  1. Truncate the table to be restored by running the cql command TRUNCATE TABLE [keyspace_name.table_name]
  2. Load the table data from the snapshot by running the cql command
    sstableloader -d <hostname_or_ip_seed_node> -u <username> -p <password> <snapshot_for_table_directory>
Note   The path of the directory snapshot_for_table_directory must end with <keyspace>/<table>, because the tool uses this directory structure to find what is the table name and to which keyspace it belogs to.

Back up Apache Cassandra configuration files

Ensure that you back up the CASSANDRA_HOME/conf directory on all nodes. 

Back up API Gateway group configuration data

Ensure that you back up the API Gateway group configuration data in apigateway/groups/<group-name>/conf directory. This contains the API GatewayAPI Manager, and KPS configuration data. 

Related Links