Trigger Jenkins Job using AWS Lambda triggered by S3 Event

The problem that we faced:

There was a cron scheduled in our environment at a particular time which used to run at 8AM. This cron read data from the s3 bucket and used to succeed but failed when there were no objects present.

Problem this Integration solves:

  • Eliminates the need of keeping a scheduler in the system. Lambda will work as a Scheduler.
  • No manual intervention will be required in case the file from one service is delayed by a minute or an hour due to any reason. 
  • Data exchange and processing between teams become more efficient and less error-prone.
  • Eliminates the need of checking the presence of files on S3. It all gets automated.

Workflow:

  1. The object is uploaded to S3
  2. S3 event triggers a Lambda function
  3. Lambda starts Jenkins job using Jenkins API
Continue reading “Trigger Jenkins Job using AWS Lambda triggered by S3 Event”

Kubernetes CSI: Container Storage Interface – Part 1

There are different application categories in the general application world, but we usually define them in two major types, i.e., stateless and stateful applications.

To have a clearer perspective, we can say that API-based applications are generally stateless, and databases are stateful. In simple words or definition, a stateless application is an application that doesn’t save or persists the client data. On the other hand, a stateful application saves data about each client and uses it for other requests.

In the older days, when we didn’t have a concept of containers and container orchestrators, there was a common way to manage both types of applications: server. For example- API and database-based applications get hosted on servers but with different configurations depending on the resource requirements.

Continue reading “Kubernetes CSI: Container Storage Interface – Part 1”

Cross Region Internal Load Balancing  in AWS with VPC Peering

Load Balancing is a method of addressing High Availability in any Cloud deployment. Load Balancers note the health of backend resources, thereby not sending traffic to servers that are not able to fulfill requests. The main aim of load balancing is to prevent any single server from getting overloaded and possibly breaking down.

In this blog, we are talking about internal load balancing with Network Load Balancer and Application Load Balancer. Network Load Balancer automatically provides a static IP per Availability Zone (subnet) to be used by applications as the front-end IP of the Load Balancer.

Continue reading “Cross Region Internal Load Balancing  in AWS with VPC Peering”

AWS Gateway LoadBalancer: A Load Balancer that we deserve

Nowadays, LoadBalancing is one of the basic needs for the application systems to perform optimally while considering some important factors like- scalability and high availability. Every cloud is providing LBaaS (LoadBalancing as a Service) as an offering so the consumers don’t have to worry about the setup and management of load-balancers by themselves.

But it’s not like that cloud is offering a single type of load balancer for every use case because for different use-case we require a different type of load balancer. For example- we have different load-balancers for Layer4 and Layer7 level traffic.

Recently AWS had a new family member in their load-balancer family and they named it “Gateway Load Balancer“. So gateway load-balancer is a load-balancing service provided by AWS to send traffic to the different appliances, applications, firewalls, etc. that are not part of the current VPC.

Continue reading “AWS Gateway LoadBalancer: A Load Balancer that we deserve”

Know How to Access S3 Bucket without IAM Roles and Use Cases

We all have used IAM credentials to access our S3 buckets. But it’s not a very safe or recommended practice to keep our Access keys and Secrets stored in a server or hard code them in our codebase.
Even if we have to use keys, we must have some mechanism in place to rotate the keys very frequently (eg: using Hashicorp Vault). Another widely adopted method is to use IAM roles attached on the EC2 instance or the AWS service accessing the bucket.

But, what if we need access to the bucket from an on-premise Data Center where we can not attach an IAM role?

Yes, we can obviously use IAM credentials and secret tokens with the rotating mechanism. But setting up the key rotation mechanism itself could be another overhead if we do not have one already in place. What if we do not require keys or roles without making the bucket public?

In this blog, I will make an attempt to cater to this problem with another alternate and easy solution.

Continue reading “Know How to Access S3 Bucket without IAM Roles and Use Cases”