This is a guide to deploy a Scylla Cluster in a generic Kubernetes environment, meaning that Scylla will not be deployed with the ideal performance. Scylla performs the best when it has fast disks and direct access to the cpu. This requires some extra setup, which is platform-specific. For specific configuration and setup, check for details about your particular environment:
- A Kubernetes cluster
- A Storage Class to provision PersistentVolumes.
- Helm 3 installed, Go to the helm docs if you need to install it. Make sure that you enable the stable repository
Running kubernetes locally is a daunting and error prone task. Fortunately there are ways to make life easier and Minikube makes it a breeze.
We need to give minikube a little bit more resources than default so start minikube like this:
minikube start --cpus=6Then make kubectl aware of this local installation like this:
eval $(minikube docker-env)In this guide you will be using the examples and manifests from Scylla Operator repository, so start off by cloning it to your local machine.
git clone git@github.com:scylladb/scylla-operator.git
cd scylla-operatorFirst deploy Cert Manager, you can either follow upsteam instructions or use following command:
kubectl apply -f examples/common/cert-manager.yamlThis will install Cert Manager to provision a self-signed certificate.
Once it's deployed, wait until Cert Manager is ready:
kubectl wait --for condition=established crd/certificates.cert-manager.io crd/issuers.cert-manager.io
kubectl -n cert-manager rollout status deployment.apps/cert-manager-webhookDeploy the Scylla Operator using the following commands:
kubectl apply -f deploy/operator.yamlThis will install the operator in namespace scylla-operator.
Wait until it's ready:
kubectl wait --for condition=established crd/scyllaclusters.scylla.scylladb.com
kubectl -n scylla-operator rollout status deployment.apps/scylla-operatorIf you want to check the logs of the operator you can do so with:
kubectl -n scylla-operator logs deployment.apps/scylla-operatorNow that the operator is running, we can create an instance of a Scylla cluster by creating an instance of the clusters.scylla.scylladb.com resource.
Some of that resource's values are configurable, so feel free to browse cluster.yaml and tweak the settings to your liking.
Full details for all the configuration options can be found in the Scylla Cluster CRD documentation.
When you are ready to create a Scylla cluster, simply run:
kubectl create -f examples/generic/cluster.yamlWe can verify that a Kubernetes object has been created that represents our new Scylla cluster with the command below. This is important because it shows that has successfully extended Kubernetes to make Scylla clusters a first class citizen in the Kubernetes cloud-native environment.
kubectl -n scylla get ScyllaClusterChecking the pods that are created is as easy as:
kubectl -n scylla get podsThe output should be something like:
NAME READY STATUS RESTARTS AGE
simple-cluster-us-east-1-us-east-1a-0 2/2 Running 0 9m49s
simple-cluster-us-east-1-us-east-1a-1 2/2 Running 0 7m43s
simple-cluster-us-east-1-us-east-1a-2 2/2 Running 0 6m46sIt is important to note that the operator creates these instances according to a pattern.
This pattern is as follows: CLUSTER_NAME-DATACENTER_NAME-RACK_NAME-INSTANCE_NUMBER as specified in cluster.yaml.
In the above example we have the following properties:
- CLUSTER_NAME:
simple-cluster - DATACENTER_NAME:
us-east-1 - RACK_NAME:
us-east-1a - INSTANCE_NUMBER: An automatically generated number attached to the pod name.
We picked the names to resemble something you can find in a cloud service but this is inconsequential, they can be set to anything you want.
To check if all the desired members are running, you should see the same number of entries from the following command as the number of members that was specified in cluster.yaml:
kubectl -n scylla get pod -l app=scyllaYou can also track the state of a Scylla cluster from its status. To check the current status of a Cluster, run:
kubectl -n scylla describe ScyllaCluster simple-clusterChecking the logs of the running scylla instances can be done like this:
kubectl -n scylla logs simple-cluster-us-east-1-us-east-1a-0 scyllaTo squeeze the most out of your deployment it is sometimes necessary to employ host networking.
To enable this the CRD allows for specifying a network parameter as such:
version: 5.4.3
agentVersion: 3.2.6
cpuset: true
network:
hostNetworking: trueThis will result in hosts network to be used for the Scylla Stateful Set deployment.
Sometimes it is necessary to run the container with different kernel parameters.
In order to support this, the Scylla Operator defines a cluster property sysctls that is a list of the desired key-value pairs to set.
For example: To increase the number events available for asynchronous IO processing in the Linux kernel to N set sysctls tofs.aio-max-nr=N.
spec:
sysctls:
- "fs.aio-max-nr=2097152"The operator is also capable of deploying Alternator instead of the regular Scylla.
This requires a small change in the cluster definition.
Change the cluster.yaml file from this:
spec:
agentVersion: 3.2.6
version: 5.4.3
developerMode: true
datacenter:
name: us-east-1to this:
spec:
version: 5.4.3
alternator:
port: 8000
writeIsolation: only_rmw_uses_lwt
agentVersion: 3.2.6
developerMode: true
datacenter:
name: us-east-1You can specify whichever port you want.
You must provide desired write isolation, supported values are: "always", "forbid_rmw", "only_rmw_uses_lwt". Difference between those isolation levels can be found in Scylla Alternator documentation.
Once this is done the regular CQL ports will no longer be available, the cluster is a pure Alternator cluster.
- From kubectl:
To get a cqlsh shell in your new Cluster:
kubectl exec -n scylla -it simple-cluster-us-east-1-us-east-1a-0 -- cqlsh
> DESCRIBE KEYSPACES;- From inside a Pod:
When you create a new Cluster, automatically creates a Service for the clients to use in order to access the Cluster.
The service's name follows the convention <cluster-name>-client.
You can see this Service in your cluster by running:
kubectl -n scylla describe service simple-cluster-clientPods running inside the Kubernetes cluster can use this Service to connect to Scylla. Here's an example using the Python Driver:
from cassandra.cluster import Cluster
cluster = Cluster(['simple-cluster-client.scylla.svc'])
session = cluster.connect()If you are running the Alternator you can access the API on the port you specified using plain http.
The operator can take a ConfigMap and apply it to the scylla.yaml configuration file.
This is done by adding a ConfigMap to Kubernetes and refering to this in the Rack specification.
The ConfigMap is just a file called scylla.yaml that has the properties you want to change in it.
The operator will take the default properties for the rest of the configuration.
- Create a ConfigMap the default name that the operator uses is
scylla-config:
kubectl create configmap scylla-config -n scylla --from-file=/path/to/scylla.yaml- Wait for the mount to propagate and then restart the cluster:
kubectl rollout restart -n scylla statefulset/simple-cluster-us-east-1-us-east-1a- The new config should be applied automatically by the operator, check the logs to be sure.
Configuring cassandra-rackdc.properties is done by adding the file to the same mount as scylla.yaml.
kubectl create configmap scylla-config -n scylla --from-file=/tmp/scylla.yaml --from-file=/tmp/cassandra-rackdc.properties -o yaml --dry-run | kubectl replace -f -The operator will then apply the overridable properties prefer_local and dc_suffix if they are available in the provided mounted file.
:::{note}
If you want to enable authentication, you first need to adjust system_auth keyspace replication factor to the number of nodes in the datacenter via cqlsh. It allows you to ensure that the user’s information is kept highly available for the cluster. If system_auth is not equal to the number of nodes and a node fails, the user whose information is on that node will be denied access.
For production environments only use NetworkTopologyStrategy.
kubectl -n scylla exec -it pods/simple-cluster-us-east-1-us-east-1a-0 -c scylla -- cqlsh -e "ALTER KEYSPACE system_auth WITH REPLICATION = {'class' : 'NetworkTopologyStrategy', 'us-east-1' : <replication_factor>};"You can read more about enabling authentication in the Enable authentication section of ScyllaDB's documentation. :::
The operator creates a second container for each scylla instance that runs Scylla Manager Agent. This container serves as a sidecar and it's the main endpoint for interacting with Scylla API. The Scylla Manager Agent can be configured with various things such as the security token used to allow access to Scylla API and storage providers for backups.
To configure the agent you just create a new secret called scylla-agent-config-secret and populate it with the contents in the scylla-manager-agent.yaml file like this:
kubectl create secret -n scylla generic scylla-agent-config-secret --from-file scylla-manager-agent.yamlSee Scylla Manager Agent configuration for a complete reference of the Scylla Manager agent config file.
Operator provisions Agent auth token by copying value from user provided config secret or auto generates it if it's empty.
To check which value is being used, decode content of <cluster-name>-auth-token secret.
To change it simply remove the secret. Operator will create a new one. To pick up the change in the cluster, initiate a rolling restart.
To set up monitoring using Prometheus and Grafana follow this guide.
The operator supports adding new nodes to existing racks, adding new racks to the cluster, as well as removing both single nodes and entire racks. To introduce the changes, edit the cluster with:
kubectl -n scylla edit scyllaclusters.scylla.scylladb.com/simple-cluster- To modify the number of nodes in a rack, update the
membersfield of the selected rack to a desired value. - To add a new rack, append it to the
.spec.datacenter.rackslist. Remember to choose a unique rack name for the new rack. - To remove a rack, first scale it down to zero nodes, and then remove it from
.spec.datacenter.rackslist.
Having edited and saved the yaml, you can check your cluster's Status and Events to retrieve information about what's happening:
kubectl -n scylla describe scyllaclusters.scylla.scylladb.com/simple-cluster:::{note}
If you have configured ScyllaDB with authenticator set to PasswordAuthenticator, you need to manually configure the replication factor of the system_auth keyspace with every scaling operation.
kubectl -n scylla exec -it pods/simple-cluster-us-east-1-us-east-1a-0 -c scylla -- cqlsh -u <username> -p <password> -e "ALTER KEYSPACE system_auth WITH REPLICATION = {'class' : 'NetworkTopologyStrategy', 'us-east-1' : <new_replication_factor>};"It is recommended to set system_auth replication factor to the number of nodes in each datacenter.
:::
After deploying our cluster along with the monitoring, we can benchmark it using cassandra-stress and see its performance in Grafana. We have a mini cli that generates Kubernetes Jobs that run cassandra-stress against a cluster.
Because cassandra-stress doesn't scale well to multiple cores, we use multiple jobs with a small core count for each
# Run a benchmark with 10 jobs, with 6 cpus and 50.000.000 operations each.
# Each Job will throttle throughput to 30.000 ops/sec for a total of 300.000 ops/sec.
hack/cass-stress-gen.py --num-jobs=10 --cpu=6 --memory=20G --ops=50000000 --limit=30000
kubectl apply -f scripts/cassandra-stress.yamlMake sure you set the proper arguments in case you have altered things such as name or namespace.
./hack/cass-stress-gen.py -h
usage: cass-stress-gen.py [-h] [--num-jobs NUM_JOBS] [--name NAME] [--namespace NAMESPACE] [--scylla-version SCYLLA_VERSION] [--host HOST] [--cpu CPU] [--memory MEMORY] [--ops OPS] [--threads THREADS] [--limit LIMIT]
[--connections-per-host CONNECTIONS_PER_HOST] [--print-to-stdout] [--nodeselector NODESELECTOR]
Generate cassandra-stress job templates for Kubernetes.
optional arguments:
-h, --help show this help message and exit
--num-jobs NUM_JOBS number of Kubernetes jobs to generate - defaults to 1
--name NAME name of the generated yaml file - defaults to cassandra-stress
--namespace NAMESPACE
namespace of the cassandra-stress jobs - defaults to "default"
--scylla-version SCYLLA_VERSION
version of scylla server to use for cassandra-stress - defaults to 4.0.0
--host HOST ip or dns name of host to connect to - defaults to scylla-cluster-client.scylla.svc
--cpu CPU number of cpus that will be used for each job - defaults to 1
--memory MEMORY memory that will be used for each job in GB, ie 2G - defaults to 2G * cpu
--ops OPS number of operations for each job - defaults to 10000000
--threads THREADS number of threads used for each job - defaults to 50 * cpu
--limit LIMIT rate limit for each job - defaults to no rate-limiting
--connections-per-host CONNECTIONS_PER_HOST
number of connections per host - defaults to number of cpus
--print-to-stdout print to stdout instead of writing to a file
--nodeselector NODESELECTOR
nodeselector limits cassandra-stress pods to certain nodes. Use as a label selector, eg. --nodeselector role=scyllaWhile the benchmark is running, open up Grafana and take a look at the monitoring metrics.
After the Jobs finish, clean them up with:
kubectl delete -f scripts/cassandra-stress.yamlTo clean up all resources associated with this walk-through, you can run the commands below.
NOTE: this will destroy your database and delete all of its associated data.
kubectl delete -f examples/generic/cluster.yaml
kubectl delete -f deploy/operator.yaml
kubectl delete -f examples/common/cert-manager.yamlIf the cluster does not come up, the first step would be to examine the operator's logs:
kubectl -n scylla-operator logs deployment.apps/scylla-operatorIf everything looks OK in the operator logs, you can also look in the logs for one of the Scylla instances:
kubectl -n scylla logs simple-cluster-us-east-1-us-east-1a-0