Multi-AZ Cassandra

Cassandra Failover and Consistency

Apache Cassandra is the always-on NoSQL database that is highly scalable and available. That sounds magical and is in fact true – IF you understand how to configure it correctly ! This article describes an issue we ran into when setting up a multi-DC configuration for Cassandra failover and how it was resolved.

Cassandra Configuration

Single Region, Dual AZ

The diagram below shows the initial system configuration for a cluster deployed across two availability zones (AZ) in the US-East region of AWS.

Multi-AZ Cassandra

We  configured Cassandra to use multiple DataCenters with each AZ being in one DC. The replication factor was set to 3. Cassandra ensures that at least one replica of each partition will reside across the two data centers.

Thus the dual AZs provide some protection against failures –  if one AZ went down, the database would still be up. 

But what happens if the entire US-East Region became unavailable? An always-on database that is accessed from anywhere in the country should be able to survive a full region failure (or most likely network failure to the region).

Cassandra Failover using Multi-Region, Multi-AZ

We modified the above configuration for protection against region failure but without adding any additional nodes. In future, we can enhance it with additional nodes as required. The beauty of Cassandra is that you do not have to over-provision infrastructure – nodes can be added dynamically without stopping operations (albeit carefully!)

Shown below is the multi-region configuration. The definition of the DC was changed. We still use 2 DCs but now all 4 nodes in US-East are in a single DC while the 2 nodes in US-West are in the second DC.

Client Configuration

Contact Points

In a Multi-DC configuration, it is important to ensure that clients are connecting to the “local” DC by default. You do this by using a load balancing policy that is both token-aware and DC-aware. For example, if a web application is running in US-East, you want it to access the Cassandra nodes in the US-East DC. This will ensure that it doesn’t suffer from latency by going across the country in its default mode. You can specify the “contact points”  for the client to connect to as well – it is a good idea to specify at least one node in each DC. An example to do this programmatically is shown below:

Cluster cluster = Cluster.builder()
    .addContactPoint(new InetSocketAddress("", 9042))
    .addContactPoint(new InetSocketAddress("", 9042))
.withLoadBalancingPolicy(new TokenAwarePolicy(
new DCAwareRoundRobinPolicy(localDC,
usedHostsPerRemoteDC, allowRemoteDCsForLocalConsistencyLevel

These settings can also be configured in the application.conf file. See the Java Driver documentation for details.

Consistency Level

The default Consistency Level in Cassandra is ONE i.e. when a query reads or writes a partition, it is sufficient for one of the replicas to acknowledge the query. This level is insufficient for most non-trivial applications; especially not for applications deploying across multiple data centers that require high availability.

In general, a consistency level of QUORUM applied to both reads and writes ensures that a Cassandra database is consistent. In our example, this implies that at least 2 replicas needs to acknowledge completion of the transaction. 

When a cluster is deployed across multiple regions, QUORUM could be problematic as it requires the coordinator to constantly send requests across regions that can cause significant latencies. To avoid performance issues, it is recommended to use LOCAL_QUORUM i.e. a quorum of nodes within the “local” DC is used to maintain consistency. Therefore, we configured the consistency level to be LOCAL_QUORUM.

Cassandra Failover Test

Now that the cluster and clients were configured, the next step was to do a Cassandra failover test. That is, we want to ensure that if all the US-East nodes failed, Cassandra would automatically failover to the US-West DC and clients would be blissfully unaware !

All the nodes in US-East were brought down; running a query after this failed with the following error:

Not enough replicas available for query at consistency LOCAL_QUORUM.

The LOCAL_QUORUM consistency level while performant, does not allow for the query to switch DCs. If the requirement is to have a transparent failover while maintaining a consistency across 2 nodes, then a better strategy is to use a consistency level of 2. With a level of 2, if the client cannot contact any of the local hosts, it will automatically switch to the remote DC.


Cassandra is an always-on database that can provide tremendous scalability. However, ensuring that it remains always-on requires that it be configured correctly while also maintaining its high performance.


Data Aces has experience developing and managing Apache Cassandra and DataStax Enterprise deployments. Please contact us for more information.

Naren Gokul

Cqlsh with SSL: Securing Cassandra Cqlsh

At Data-Aces, we provide managed services running Cassandra for our customers. With more and more attention being given to data security these days, end-to-end encryption of ALL communication is becoming a mandatory requirement. This includes traffic between the Cassandra nodes or from client to the Cassandra cluster. This means, cqlsh with SSL is also a requirement. The default config file (Cassandra.yaml) is configured for normal, unencrypted communication between clients and Cassandra on port 9042.

It is fairly easy to setup client to cluster encryption by creating certificates and adding them to Java keystore/truststore. This process is documented here:

As described in the link above, this involves changing ‘enabled’ to ‘true’ in the following section of the Cassandra.yaml file:
enabled: true
# If enabled and optional is set to true encrypted and unencrypted connections are handled.
optional: false
This will ensure that ALL client connections will be encrypted even if still using port 9042.  Note that after the above changes, you will no longer be able to connect to the database using the CQL command line tool cqlsh!
The Cassandra documentation provides basic information as to how to setup cqlsh with SSL.

However, following these steps, I couldn’t get cqlsh with SSL to work in our cluster. This is because, it refers to certfile without giving us an important caveat: cqlsh is a python application and does not use the Java keystore/truststore setup for normal Cassandra and Java clients.

Instead, I had to convert the certificate in ‘keystore.jks’ to PKCS12 format. Luckily, keytool has an inbuilt facility to convert to PKCS12 format.

keytool -importkeystore -srckeystore keystore -destkeystore pkcs12ks -deststoretype PKCS12 -srcstorepass keystorepassword -deststorepass keystorepassword

This will create a keystore file named ‘pkcs12ks’ in the current directory.

The next step is to create a PEM file from this keystore as follows:

openssl pkcs12 -in pkcs12ks -nokeys -out cqlsh.pem -passin pass:keystorepassword 

Setting up .cqlshrc file

Now that we have our PEM file, we can point to it in the cqlshrc file

factory = cqlshlib.ssl.ssl_transport_factory 
certfile = /home/user/cqlsh.pem validate = false

Note that setting ‘validate=true’ implies that the certificate needs to be validated at connection time.

We are now all set to run cqlsh with SSL using the option –ssl as below:

$ cqlsh <host_ip> --ssl