Enable Support to Provision GP3 Volumes in Storage Class

No matter where organizations are in their cloud journey, cloud cost optimization remains an essential and top-priority concern. As organizations align increasing workloads to the cloud in terms of size there is a high possibility to lose sight of the overall cloud environment and the costs associated with it. Therefore, it’s essential to optimize cloud costs so as to maximise return over investment (ROI). With the same concern we were working towards reducing the cost of provisioned storage for one our clients. This post covers a particularly interesting issue around the same.

When we found that gp3 volumes are cheaper than gp2 by looking at price comparison given by aws, we decided to migrate. Once our migration was complete w.r.t. volumes under cost optimization and mail was circulated as well stating that we would be launching only gp3 ebs volumes from now onwards, we did not expect one particular issue to creep up on us. After 2-3 days, we saw that gp2 volumes were still launching.

We checked the volumes that were launched recently and found the culprit “storage class”. We were using a storage class for dynamic volume provisioning of aws ebs volumes. Turns out, we forgot to change storage class in our eks cluster.

Warning ProvisioningFailed 9s (x5 over 59s) persistentvolume-controller Failed to provision volume with StorageClass "gp3": invalid AWS VolumeType "gp3".

With the above error we came to know that we need to change the underlying provisioner of the storage class because provisioner: kubernetes.io/aws-ebs only supported gp2 volume type as per official documentation. On further Googling to get out of this sticky situation, we found a CSI provisioner could fix this problem. But implementing it wasn’t exactly a straight shot.

Ask me ?

Problem Statement

Unable to launch gp3 volumes with in-tree kubernetes plugin. Followed a blog, but ended up with another error, “not able to provision persistent volume using csi provisioner”

Steps to reproduce


Delete storage class with default provisioner i.e. Fig. 1 below
kubectl delete sc buildpiper-storage

Create storage class with csi provisioner i.e. Fig. 2 below
kubectl apply -f storage-class-csi.yaml

Get eks worker node role name i.e. Fig. 3 below
kubectl get configmap aws-auth -n kube-system

copy the ROLE-NAME
create IAM policy with name Amazon_EBS_CSI_Driver

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "ec2:AttachVolume",
        "ec2:CreateSnapshot",
        "ec2:CreateTags",
        "ec2:CreateVolume",
        "ec2:DeleteSnapshot",
        "ec2:DeleteTags",
        "ec2:DeleteVolume",
        "ec2:DescribeInstances",
        "ec2:DescribeSnapshots",
        "ec2:DescribeTags",
        "ec2:DescribeVolumes",
        "ec2:DetachVolume"
      ],
      "Resource": "*"
    }
  ]
}

Associate IAM Policy to Worker Node IAM Role:
1. Open AWS console -> Go to Services -> IAM -> Roles

2. Search for role with name  ROLE-NAME  and open it

3. Click on Permissions tab – Click on Attach Policies

4. Search for Amazon_EBS_CSI_Driver and click on Attach Policy

Install EBS CSI Driver
kubectl apply -k "github.com/kubernetes-sigs/aws-ebs-csi-driver/deploy/kubernetes/overlays/stable/?ref=release-0.1

Fig. 1: Storage class with default provisioner and type

Fig. 2: storage class i.e. storage-class-csi.yaml with csi provisoner and type gp3

Fig. 3: aws-auth configmap

Resolution

Continuing to struggle with this issue, we looked up compatible version at Kubernetes Version Compatibility Matrix. Following this, upgraded k8s cluster from version 1.16 to 1.17 and it started working with version “release-0.10” for me.

Conclusion

We sought cost optimization and during that journey, came across a completely different issue. Got to know about CSI plugin that helped to launch gp3 ebs volumes via storage class. We had also faced version compatibility issue that leads to failure in achieving intended result. Overall it was a satisfying experience for me.
I have tried to keep it simple and frame the content based on my experience. If you feel it needs any correction or feedback, feel free to comment.
Keep learning and stay safe !

Image Reference

Image 1 and Image 2,

Blog Pundit: Adeel Ahmad & Naveen Verma

Opstree is an End to End DevOps solution provider

Connect Us

Author: V!kash Gautam

DevOps Engineer

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s