Kong x Cassandra DB Restore Failing to pull old (restored) table entries

1

Greetings All,

My Kong x Cassandra journey has been an interesting one. I wanted to reach out to hopefully get some advice on Kong accessing restored data from the Cassandra snapshot. It appears I am successfully restoring the schema and data tables for Cassandra (completing ‘nodetool refresh KEYSPACE TABLE’) however, when I use Postman to access Kong API to view all services I’m getting an empty array even though I can see the populated/restored data in the table via CQLSH. I’m curious to know what required tables need to be restored at minimum to allow Kong to access the older data. I started restoring tables one by one starting with Services, Routes, Consumers etc. but still haven’t rekindled the link between Kong instance and backup data.

For additional validation I created a new service and ran another GET via postman and the new service returns in the array as well as populates in the same table with the restored data visible in CQLSH.

2

I resolved this by truncating the table instead of dropping the keyspace and importing the schema. This resolved the issue with Cassandra shadowing the old entries in the DB that were restored form the snapshot/backup. [Resource Link](Resource Link)

Note: You may not need to truncate under certain conditions. For example, if a node lost a disk, you might restart before restoring so that the node continues to receive new writes before starting the restore procedure. 

Truncating is usually necessary. For example, if there was an accidental deletion of data, the tombstone from that delete has a later write timestamp than the data in the snapshot. If you restore without truncating (removing the tombstone), the database continues to shadow the restored data. This behavior also occurs for other types of overwrites and causes the same problem.```
3

Following up on this discussion I was able to restore the Cassandra cluster to a new node however the Kong instance is still failing to pull the data that is visible via the CQL table.

Here our the steps i’m taking to produce this issue:

  1. Deploy Cassandra db Instance
  2. Deploy Kong instance configured to Cassandra cluster IP
  3. Push data (Creating new services) via Kong API
  4. Create Snapshot of Cassandra kong keyspace
  5. Export Schema for kong keyspace
  6. Export tokens for Cassandra node
  7. Tear down Cassandra instance
  8. Tear down Kong instance
  9. Create new Cassandra instance
  10. Create new Kong instance configured to new Cassandra instance via cluster IP
  11. Export cassandra.yaml, edit ‘initial_token’ key-value pair with with exported tokens from previous instance. Also configuring ‘incremental_backups’ to true
  12. Build configMap with modified cassandra.yaml and applying it to Cassandra instance via helm upgrade.
  13. Truncate kong keystore / services table
  14. Kubectl cp snapshot backup to …/services-UID directory
  15. Executing nodetool refresh kong services
  16. Verifying the data is now present via cqlsh (Cassandra)
  17. Running Kong Get Services via API returns Empty array
  18. Pushing new data to Kong (Creating services)
  19. Running Kong Get Services via APU returns new data
  20. Checking CQLSH services table shows combined old and new data visible in same table.

This has me stymied as I’m unsure what steps I could be missing to allow the restore data to reflect in the API.

I am curious as to how to restart the cassandra service and not use nodetool refresh as I have read this can cause data to be missing or overshadowed.

I am following the steps outlined here. However, I’m having issues with #4 and #7 as the directory does not exist in the pod and I am unable to determine the Cassandra service in order to stop it. I know these aren’t directly related to kong but wanted to provide a full scope.

Please advise,