Thursday, January 23, 2014

How to Manage Amazon Web Services Instances part 1

If you want to minimize the amount of money you spend on Amazon Web Services (AWS) infrastructure, then this blog post is for you. In this post I will be discussing  the rationale behind starting & stopping AWS instances in an automated fashion and more importantly, doing it in a correct way. Obviously you could do it through the web console of AWS as well, but it will need your daily involvement. In addition, you would have to take care of starting/stopping various services running on those instances.

Before directly jumping on how we achieved instance management in an automated fashion, I like to state the problem that we were facing. Our application testing infrastructure is on AWS and it is a multiple components(20+) application distributed among 8-9 Amazon instances. Usually our testing team starts working from 10 am, and continues till 7 pm. Earlier we used to keep our testing infrastructure up for 24 hours, even though we were using it for only 9 hours on weekdays, and not using it at all on weekends. Thus, we were wasting more then 50% of the money that we spent on the AWS infrastructure. The obvious solution to this problem was: we needed an intelligent system that would make sure that our amazon infrastructure was up only during the time when we needed it.

The detailed list of the requirements, and the corresponding things that we did were:
  1. We should shut down our infrastructure instances when we are not using them.
  2. There should be a functionality to bring up the infrastructure manually: We created a group of Jenkins jobs, which were scheduled to run at a specific time to start our infrastructure. Also a set of people have execution access to these jobs to start the infrastructure manually, if the need arises.
  3. We should bring up our infrastructure instances when we need it.
  4. There should be a functionality to shut down the infrastructure manually: We created a group of Jenkins jobs that were scheduled to run at a specific time to shut down our infrastructure. Also a set of people have execution access on these jobs to shut down the infrastructure manually, if the need arises.
  5. Automated application/services start on instance start: We made sure that all the applications and services were up and running when the instance was started.
  6. Automated graceful application/services shut down before instance shut down: We made sure that all the applications and services were gracefully stopped before the instance was shut down, so that there was be no loss of data.
  7. We also had to make sure that all the applications and services should be started as per defined agreed order.

Once we had the requirements ready, implementing them was simple, as Amazon provides a number of APIs to achieve this. We used AWS CLI, and needed to use just 2 simple commands that AWS CLI provides.
The command to start an instance :
aws ec2 start-instances --instance-ids i-XXXXXXXX
The command to stop an instance :
aws ec2 stop-instances --instance-ids i-XXXXXXXX 

Through above commands you can automate starting and stopping AWS instances, but you might not be doing it the correct way. As you didn't restrict the AWS CLI allow firing of start-instances and stop-instances commands only, you could use other commands and that could turn out to be a problem area. Another important point to consider is that we should restrict the AWS instances on which above commands could be executed, as these commands could be mistakenly run with the instance id of a production amazon instance id as an argument, creating undesirable circumstances :)

In the next blog post I will talk about how to start and stop AWS instances in a correct way.

8 comments:

  1. Bluehost is ultimately the best hosting company with plans for any hosting requirments.

    ReplyDelete
  2. This is excellent information. It is an amazing and wonderful to visit your blog. So, please keep posting.Aws Jobs in Hyderabad.

    ReplyDelete
  3. Hi, I have read your blog. Really very informative and excellent post I had ever seen about AWS. Thank you for sharing such a wonderful blog to our vision. Learn AWS Training in Bangalore to know more details about this technology. AWS Training in Chennai.

    ReplyDelete
  4. Thanks for this excellent article :) Keep posting.

    - Nandini,
    Trainer,
    Kamal Technologies - Best Node JS training institute in Chennai

    ReplyDelete
  5. Needed to compose you a very little word to thank you yet again regarding the nice suggestions you’ve contributed here. aws training in chennai

    ReplyDelete

  6. I've been surfing on the web more than 3 hours today, yet I never found any stupefying article like yours. It's imperatively worth for me. As I would see it, if all web proprietors and bloggers made confusing substance as you did, the net will be in a general sense more profitable than at whatever point in late memory.

    Digital Marketing Training in Mumbai

    Six Sigma Training in Dubai

    Six Sigma Abu Dhabi

    ReplyDelete
  7. I think this is a great site to post and I have read most of contents and I found it useful for my Career .Thanks for the useful information. For any information or Queries Comment like and share it.

    PMP Training Abu Dhabi

    GDPR Training in Hyderabad

    Digital Marketing Training in Hyderabad


    six sigma Training Pune

    ReplyDelete
  8. Very nice post here and thanks for it .I always like and such a super contents of these post.Excellent and very cool idea and great content of different kinds of the valuable information's.
    Good discussion. Thank you.
    Anexas
    Six Sigma Training in Abu Dhabi
    Six Sigma Training in Dammam
    Six Sigma Training in Riyadh

    ReplyDelete