Cassandra backup strategy with Kong

Installation/Setup
1

We’ve just begun using Kong EE and our first sandbox implementation has Kong and Cassandra co-resident on a single VM. Our plan is to evolve to a Cassandra cluster, and I was looking for some feedback on Cassandra care-n-feeding in that model (this is the first use of Cassandra in our organization, so breaking new ground here).

At a basic level, what is a reasonable strategy for DB backup/restore when using a Cassandra cluster? Assuming a properly deployed cluster provides for resiliency, I would expect we would still need some recovery procedure in case of DB data corruption which impacts the entire cluster. Originally we thought that using cURL scripts to define all the Kong metadata would be sufficient … if the DB becomes corrupted, then drop the Kong keyspace, let Kong reconstitute the schema it needs with “migrations up”, then recreate all the objects (services, routes, etc.) from the cURL scripts. But now not sure if that’s really the best approach.

So wanted to get some feedback on how others tackle this. Thanks in advance!

2

We use https://github.com/gamunu/cassandra_snap for Cassandra backups(Uses S3 backups) and restore, I have a few pending PR fixes to that repo too.

Also we run a few helper bash scripts to do restoration logic using cassandra_snap too,

bash restoreNodesV2.sh root host1,host2,host3 kong-cassandra-backups company-core-1 kong-cassandra-dev keyId key 20180510214836 /data/tmp/cassandra_restore/ kong_dev /data/cassandra/data/

bash restoreNodesV3.sh root restoreToHost1,restoreToHost2 restoreFromHost1,restoreFromHost2 kong-cassandra-backups company-core-1 kong-cassandra-dev keyId key 20180510214836 /data/tmp/cassandra_restore/ kong_dev /data/cassandra/data/

restoreNodesV2 lets you restore data from same host to same host

#!/bin/bash
restoreNodesV2()
{
    local username="${1}"
    local hostnames="${2}"
    local s3bucketname="${3}"
    local s3bucketregion="${4}"
    local s3basepath="${5}"
    local awsaccesskeyid="${6}"
    local awssecretaccesskey="${7}"
    local snapshotname="${8}"
    local restoredir="${9}"
    local keyspace="${10}"
    local cassandradatadir="${11}"
    
    for hostname in $(echo $hostnames | sed "s/,/ /g")
    do
       if ssh ${username}@${hostname} "echo 2>&1"; then
        ssh ${username}@${hostname} "service cassandra stop"
        ssh ${username}@${hostname} "cassandra-snapshotter --s3-bucket-name=${s3bucketname} --s3-bucket-region=${s3bucketregion} --s3-base-path=${s3basepath} --aws-access-key-id=${awsaccesskeyid} --aws-secret-access-key=${awssecretaccesskey} --verbose restore --hosts=${hostname} --snapshot-name ${snapshotname} --restore-dir ${restoredir} --keyspace ${keyspace} --local"
        ssh ${username}@${hostname} "chmod -R 755 ${restoredir}${keyspace}/"
        ssh ${username}@${hostname} "chown -R cassandra:cassandra ${restoredir}${keyspace}/"
        ssh ${username}@${hostname} "rm -rf ${cassandradatadir}${keyspace}/*"
        ssh ${username}@${hostname} "rsync -a ${restoredir}${keyspace}/* ${cassandradatadir}${keyspace}/"
        sleep 5
        ssh ${username}@${hostname} "service cassandra start"
        sleep 5
        echo "Success: "${hostname}
       else
        echo "Failed: "${hostname}
       fi
    done
}

restoreNodesV2 "${1}" "${2}" "${3}" "${4}" "${5}" "${6}" "${7}" "${8}" "${9}" "${10}" "${11}"

restoreNodesV3 lets you restore data from any host to any host(slight improvement I guess)

#!/bin/bash
restoreNodesV3()
{
    local username="${1}"
    IFS=',' read -r -a restoreToHosts <<< "${2}"
    IFS=',' read -r -a restoreFromHosts <<< "${3}"
    local s3bucketname="${4}"
    local s3bucketregion="${5}"
    local s3basepath="${6}"
    local awsaccesskeyid="${7}"
    local awssecretaccesskey="${8}"
    local snapshotname="${9}"
    local restoredir="${10}"
    local keyspace="${11}"
    local cassandradatadir="${12}"

    local restoreToHostsLength=${#restoreToHosts[@]}
    local restoreFromHostsLength=${#restoreFromHosts[@]}

    if [ ${restoreToHostsLength} -ne ${restoreFromHostsLength} ]; then
      echo "Error: restoreToHosts not same size as restoreFromHosts"
      exit 1
    fi

    for (( i=1; i<${restoreToHostsLength}+1; i++ ));
    do
      if ssh ${username}@${restoreToHosts[$i-1]} "echo 2>&1"; then
         ssh ${username}@${restoreToHosts[$i-1]} "service cassandra stop"
         ssh ${username}@${restoreToHosts[$i-1]} "cassandra-snapshotter --s3-bucket-name=${s3bucketname} --s3-bucket-region=${s3bucketregion} --s3-base-path=${s3basepath} --aws-access-key-id=${awsaccesskeyid} --aws-secret-access-key=${awssecretaccesskey} --verbose restore --hosts=${restoreFromHosts[$i-1]} --snapshot-name ${snapshotname} --restore-dir ${restoredir} --keyspace ${keyspace} --local"
         ssh ${username}@${restoreToHosts[$i-1]} "chmod -R 755 ${restoredir}${keyspace}/"
         ssh ${username}@${restoreToHosts[$i-1]} "chown -R cassandra:cassandra ${restoredir}${keyspace}/"
         ssh ${username}@${restoreToHosts[$i-1]} "rm -rf ${cassandradatadir}${keyspace}/*"
         ssh ${username}@${restoreToHosts[$i-1]} "rsync -a ${restoredir}${keyspace}/* ${cassandradatadir}${keyspace}/"
         sleep 5
         ssh ${username}@${restoreToHosts[$i-1]} "service cassandra start"
         sleep 5
         echo "Success: "${restoreToHosts[$i-1]}
      else
         echo "Failed: "${restoreToHosts[$i-1]}
      fi
    done
}

restoreNodesV3 "${1}" "${2}" "${3}" "${4}" "${5}" "${6}" "${7}" "${8}" "${9}" "${10}" "${11}"

EDIT - I do want to note if you are using EE I believe their admin UI has backups and restore built in as options, if you are a community user I recommend Konga as well to attempt Kong backup and restores - https://github.com/pantsel/konga , it may be slightly buggy but the dev is phenomenal and always looks into well documented problems with a sample way to produce error.

3

Thanks for the info!

2 Likes