Redis Setup on Kubernetes

Redis is a popular and opensource in-memory database that supports multiple data structures like strings, hashes, lists, and sets. But similar to other tools, we can scale standalone redis to a particular extent and not beyond that. That’s why we have a cluster mode setup in which we can scale Redis nodes horizontally and then distribute data among those nodes.

Since Kubernetes is becoming buzz technology and people are using it to manage their applications, databases, and middlewares at a single place. So in this blog, we will see how we can deploy the Redis cluster in production mode in the Kubernetes cluster and test failover.

But if you want to deploy the redis cluster on VMs or bare metal, you can use ansible role developed by us for redis cluster/standalone mode.

https://galaxy.ansible.com/opstree_devops/redis

Challenges with Kubernetes

Kubernetes has made the deployment of stateful application quite easy by StatefulSets. By using StatefulSets, we can easily deploy and scale any kind of stateful applications like Kafka, Zookeeper, etc.
But in the case of redis, the setup is not straightforward, there are some additional things which needs to be taken care:-

  • We have to use the headless service of Redis because it’s a TCP based service and normal service is HTTP(Layer 7) based Loadbalancer. So in case of headless service, no ClusterIP will be used and we have to rely on Pod IP.
  • Redis doesn’t use DNS to form clusters instead of that it uses IP. So we cannot use the internal DNS name of headless service, instead of that, we have to use Pod IP to form Redis cluster.
  • In Kubernetes, Pod IP is dynamic and it can change after the pod restart, so in case of the restart the cluster will be malformed and the restarted pod will act as a lost node.

We know what you guys are thinking that life is not that easy with Kubernetes as well but we have tried to make it bearable by designing a solution that can deploy and manage the Redis cluster on Kubernetes within the minutes.

The Solution

As a solution, we have developed our in-house CRD(Custom Resource Definition) to deploy and manage Redis in standalone/cluster mode. So CRD is an amazing feature of Kubernetes which allows us to create our own resources and APIs in Kubernetes. We are not going in the depth of the CRD but soon we will write a blog on CRD as well. Till that time you guys can read about CRD from the official documentation.

The API name which we have created is “redis.opstreelabs.in/v1alpha1” and this operator is also published under the OperatorHub catalog.
https://operatorhub.io/operator/redis-operator

So for deploying the redis-operator and setup we need a Kubernetes cluster 1.11+ and that’s it. Let’s deploy the redis operator first.

# Deploy the CRD API
$ kubectl apply -f https://github.com/OT-CONTAINER-KIT/redis-operator/raw/master/deploy/crds/redis.opstreelabs.in_redis_crd.yaml
# Create operator serviceaccount
$ kubectl apply -f https://github.com/OT-CONTAINER-KIT/redis-operator/raw/master/deploy/service_account.yaml
# Create operator role for access
$ kubectl apply -f https://github.com/OT-CONTAINER-KIT/redis-operator/raw/master/deploy/role.yaml
# Create rolebindings for binding the role and serviceaccount
$ kubectl apply -f https://github.com/OT-CONTAINER-KIT/redis-operator/raw/master/deploy/role_binding.yaml
# Deploy the operator deployment
$ kubectl apply -f https://github.com/OT-CONTAINER-KIT/redis-operator/raw/master/deploy/operator.yaml

Once the operator is deployed and running fine, we can deploy the Redis cluster.

---
apiVersion: redis.opstreelabs.in/v1alpha1
kind: Redis
metadata:
  name: redis
spec:
  mode: cluster
  size: 3
  global:
    image: opstree/redis:v2.0
    imagePullPolicy: IfNotPresent
    password: "Opstree@1234"
    resources:
      requests:
        cpu: 100m
        memory: 128Mi
      limits:
        cpu: 100m
        memory: 128Mi
  master:
    service:
      type: ClusterIP
  slave:
    service:
      type: ClusterIP
  redisExporter:
    enabled: true
    image: quay.io/opstree/redis-exporter:1.0
    imagePullPolicy: Always
    resources:
      requests:
        cpu: 100m
        memory: 128Mi
      limits:
        cpu: 100m
        memory: 128Mi
  storage:
    volumeClaimTemplate:
      spec:
        storageClassName: csi-cephfs-sc
        accessModes: ["ReadWriteOnce"]
        resources:
          requests:
            storage: 1Gi
      selector: {}

In the above-defined manifest, we are deploying the Redis 3 node cluster with persistence.
The manifest which are using to deploy redis is explained here

https://ot-container-kit.github.io/redis-operator/#/configuration/configuration

Note:- Make sure you change the default values as per your requirement like Resources, Storage.

# Deploy the cluster of Redis
$ kubectl apply -f redis-cluster.yaml

Then validate the Redis cluster is working fine or not.

$ kubectl exec -it redis-master-0 bash
$ redis-cli -c -a Opstree@1234
$ cluster nodes

Failover Testing

Before failover testing, we have to write some dummy data inside the Redis cluster, we can write the dummy data using the redis-cli.

$ set tony stark
$ get tony

Let’s restart the pod name “redis-master-0” and see the redis node behavior.

$ kubectl delete pod redis-master-0 
$ kubectl exec -it redis-master-0 bash
$ redis-cli -c -a Opstree@1234
$ cluster nodes

So if you notice the output of cluster nodes command, the node IP is updated and it’s connected as a master. Also, the data of the “tony” key still persists in the redis cluster.

Conclusion

The redis-operator is not only limited to the Redis setup only but it also provides some other features like:-

  • Password/Password-less setup
  • Cluster/Standalone Redis setup
  • Monitoring with redis exporter and Prometheus service discovery annotations
  • Security context to manage system parameters
  • Affinity and Priority class to manage node failure
  • Database and caching mode support
  • Secure Redis image

For further information on redis-exporter setup and use cases, you can refer to the documentation page of it.
https://ot-container-kit.github.io/redis-operator/#/

If you face any issue or you want to ask any question, please feel free to use the comment section of this blog. Also, if you have some issues or feature requests regarding Redis Operator you can raise the issue for that in GitHub.
https://github.com/OT-CONTAINER-KIT/redis-operator

Thanks for reading, I’d really appreciate any and all feedback, please leave your comment below if you guys have any feedback.

Cheers till next time!!

Opstree is an End to End DevOps solution provider

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s