The closer you think you are, the less you’ll actually see

I hope you have seen the movie Now you see me, it has a famous quote The closer you think you are, the less you’ll actually see. Well, this blog is not about this movie but how I got stuck into an issue, because I was not paying attention and looking at the things closely and seeing less hence not able to resolve the issue.

There is a lot happening in today’s DevOps world. And HashiCorp has emerged out to be a big player in this game. Terraform is one of the open source tools to manage infrastructure as code. It plays well with most of the cloud provider. But with all these continuous improvements and enhancements there comes a possibility of issues as well. Below article is about such a scenario. And in case you have found yourself in the same trouble. You are lucky to reach the right page.
I was learning terraform and performing a simple task to launch an Ubuntu EC2 instance in us-east-1 region. For which I required the AMI Id, which I copied from the AWS console as shown in below screenshot.

Once I got the AMI Id, I tried to create the instance using terraform, below is the screenshot of the code

provider “aws” {
  region     = “us-east-1”
  access_key = “XXXXXXXXXXXXXXXXXX”
  secret_key = “XXXXXXXXXXXXXXXXXXX”
}
resource “aws_instance” “sandy” {
        ami = “ami-036ede09922dadc9b
        instance_type = “t2.micro”
        subnet_id = “subnet-0bf4261d26b8dc3fc”
}
I was expecting to see the magic of Terraform but what I got below ugly error.

Terraform was not allowing to spin up the instance. I tried couple of things which didn’t work. As you can see the error message didn’t give too much information. Finally, I thought of giving it a try by  doing same task via AWS web console. I searched for the same ubuntu AMI and selected the image as shown below. Rest of the things, I kept to default. And well, this time it got launched.

And it confused me more. Through console, it was working fine but while using Terraform it says not allowed. After a lot of hair pulling finally, I found the culprit which is a perfect example of how overlooking small things can lead to blunder.

Culprit

While copying the AMI ID from AWS console, I had copied the 64-bit (ARM) AMI ID. Please look carefully, the below screenshot

But while creating it through console I was selecting the default configuration which by is 64-bit(x86). Look at the below screenshot.

To explain it further, I tried to launch the VM with 64-bit (ARM) manually. And while selecting the AMI, I selected the 64-bit (ARM).

And here is the culprit. 64-bit(ARM) only supports a1 instance type

Conclusion

While launching the instance with the terraform, I tried using 64-bit (ARM) AMI ID mistakenly, primarily because for same AMI there are 2 AMI IDs and it is not very visible to eyes unless you pay special attention.

So folks, next time choosing an AMI ID keep it in mind what type of AMI you are selecting. It will save you a lot of time.

Migrate your data between various Databases

Data Migration Service

 
Have you ever thought about migrating your production database from one platform to another
and dropped this idea later, because it was too risky, you were not ready to
bare a downtime?
If yes, then please pay attention because this is what we are going to perform
in this article.
A few days back we’re trying to migrate our production MySQL RDS from AWS to GCP,  SQL, and we had to migrate data without downtime, accurate and
real-time and that too without the help
of any Database Administrator.
 
After doing a bit research and evaluating few services we finally started working on AWS DMS (Data Migration Service) and figured out this is a great service to migrate a
different kind of data.
 
You can migrate your data to and from the most widely used commercial and open-source databases, and database platforms. Databases like Oracle, Microsoft SQL Server, and
PostgreSQL, MongoDB.
The source database remains fully operational during the migration,
The service supports
homogeneous migrations such as Oracle to Oracle,
and also heterogeneous migrations between different database platforms.
 

Let’s discuss some important features of AWS DMS:

 
  • Migrates the database securely, quickly and accurately.
  • No downtime required, works as schema converter as well.
  • Supports various type or database like MySQL, MongoDB, PSQL etc.
  • Migrates real-time data also synchronize ongoing changes.
  • Data validation is available to verify database.
  • Compatible with a long range of database platforms like RDS, Google SQL, on-premises etc.
  • Inexpensive (Pricing is based on the compute resources used during the migration process).
This is a typical migration scenario.
Let’s perform step by step migration:

Note: We’ve performed migration from AWS RDS
to GCP SQL, you can choose database source and
destination as per your requirement.

  1. Create replication instance:
    A replication instance initiates the connection between the source and target databases, transfers the data, cache any changes that occur on the source database during the initial data load.
    Use the fields to below to configure the parameters of your new replication instance including network and security information, encryption details, select instance class as per requirement.

    After completion, all mandatory fields click the next tab, and you will be redirected
    to Replication Instance tab.
    Grab a coffee quickly while the instance is getting ready.

    Hope you are ready with your coffee because the instance is ready now.


  2. Now we are to create two endpoints “Source” and “Target” 2.1 Create Source Endpoint:

    Click on “Run test” tab after completing all fields, make sure your Replication instance IP is whitelisted
    under security group. 2.2 Create Target Endpoint


    Click on “Run test” tab again after completing all fields, make sure your Replication instance IP is whitelisted under target DB authorization.
    Now we’ve ready Replication Instance, Source Endpoint, and Target Endpoint.
  3. Finally, we’ll create a “Replication Task” to start replication.
    Fill the fields like:
  • Task Name: any name
  • Replication Instance: The instance we’ve created above
  • Source Endpoint: The source database
  • Target Endpoint: The target database
  • Migration Type: Here I choose “Migration existing data and replication
    ongoing” because we needed ongoing changes.
 
4. Verify the task status now.
Once all the fields are completed click on the “Create task” and you will be
redirected to “Tasks”
Tab.
Check your task status
 
The task has been successfully completed now, you can verify the inserts tabs and validation tab,
The migration is done successfully if Validation State is “Validated” that means migration has been performed successfully.

Git Inside Out

Git Inside-Out
Man Wearing Black and White Stripe Shirt Looking at White Printer Papers on the Wall

Git is basically a file-system where you can retrieve your content through addresses. It simply means that you can insert any kind of data into git for which Git will hand you back a unique key you can use later to retrieve that content. We would be learning #gitinsideout through this blog

The Git object model has three types: blobs (for files), trees (for folder) and commits. 

Objects are immutable (they are added but not changed) and every object is identified by its unique SHA-1 hash
A blob is just the contents of a file. By default, every new version of a file gets a new blob, which is a snapshot of the file (not a delta like many other versioning systems).
A tree is a list of references to blobs and trees.
A commit is a reference to a tree, a reference to parent commit(s) and some decoration (message, author).

Then there are branches and tags, which are typically just references to commits.

Git stores the data in our .git/objects directory. After initialising a git repository, it automatically creates .git/objects/pack and .git/objects/info with no regular file. After pushing some files, it would reflect in the .git/objects/ folder

OBJECT Blob

blob stores the content of a file and we can check its content by command

git cat-file -p

or git show

OBJECT Tree

A tree is a simple object that has a bunch of pointers to blobs and other trees – it generally represents the contents of a directory or sub-directory.

We can use git ls-tree to list the content of the given tree object

OBJECT Commit

The “commit” object links a physical state of a tree with a description of how we got there and why.

A commit is defined by tree, parent, author, committer, comment

All three objects ( blob,Tree,Commit) are explained in details with the help of a pictorial diagram.

Often we make changes to our code and push it to SCM. I was doing it once and made multiple changes, I was thinking it would be great if I could see the details of changes through local repository itself instead to go to a remote repository server. That pushed me to explore Git more deeply.

I just created a local remote repository with the help of git bare repository. Made some changes and tracked those changes(type, content, size etc).

Below example will help you understand the concept behind it.

Suppose we have cloned a repository named kunal:

Inside the folder where we have cloned the repository, go to the folder kunal then:

cd kunal/.git/

I have added content(hello) to readme.md and made many changes into the same repository as:

adding README.md

updating Readme.md

adding 2 files modifying one

pull request

commit(adding directory).

Go to the refer folder inside .git and take the SHA value for the master head:

This commit object we can explore further with the help of cat-file which will show the type and content of tree and commit object:

Now we can see a tree object inside the tree object. Further, we can see the details for the tree object which in turn contains a blob object as below:

Below is the pictorial representation for the same:

More elaborated representation for the same :

Below are the commands for checking the content, type and size of objects( blob, tree and commit)

kunal@work:/home/git/test/kunal# cat README.md
hello

We can find the details of objects( size,type,content) with the help of #git cat-file

git-cat-file:- Provide content, type or size information for repository objects

You an verify the content of commit object and its type with git cat-file as below:

kunal@work:/home/git/test/kunal/.git # cat logs/refs/heads/master

Checking the content of a blob object(README.md, kunal and sandy)

As we can see first one is adding read me , so it is giving null parent(00000…000) and its unique SHA-1 is 912a4e85afac3b737797b5a09387a68afad816d6

Below are the details that we can fetch from above SHA-1 with the help of git cat-file :

Consider one example of merge:

Created a test branch and made changes and merged it to master.

Here you can notice we have two parents because of a merge request

You can further see the content, size, type of repository #gitobjects like:

Summary

This is pretty lengthy article but I’ve tried to make it as transparent and clear as possible. Once you work through the article and understand all concepts I showed here you will be able to work with Git more effectively.

This explanation gives the details regarding tree data structure and internal storage of objects. You can check the content (differences/commits)of the files through local .git repository which stores each object with unique  SHA  hash. This would clear basically the internal working of git.
Hopefully, this blog would help you in understanding the git inside out and helps in troubleshooting things related to git.

Log Parsing of Windows Servers on Instance Termination

 Windows
As we all know that how critical are Logs as a part of any system, they give you deep insights about your application, what your system is doing and what caused the error. Depending on how logging is configured logs may contain transaction history, timestamps and amounts debited/credited into client’s account and a lot more.

On an enterprise level application, your system goes to multiple hosts, managing the logs across multiple hosts can be complicated. Debugging the error in the application across hundreds of log files on hundreds of servers can be very time consuming and complicated and not the right approach so it is always better to move the logs to a centralized location.

 
Lately in my company I faced a situation which I assume is a very commonly faced scenario in Amazon’s Cloud where we might have to retain application logs from multiple instances behind an Auto Scaling group.  Let’s assume an example for better understanding. 
 
Suppose your application is configured to be logging into C:\Source\Application\web\logs Directory. The Application running has variant incoming traffic, sometimes it receives requests which can be handled by 2 servers, other times it may require 20 servers to handle the traffic.
 
When there is a hike in traffic, Amazon Ec2’s smart AutoScaling Group uses the configuration and scales from 2 server to many (According to ASG Policy) and during this phase, the application running in the newly launched Ec2’s also log into C:\Source\Application\web\logs …. but when there’s a drop in traffic, the ASG triggers a scale down policy, resulting to termination of instances, which also results in deletion of all the log files inside the instances launched via ASG during high traffic time.

Faced a similar situation ?  No worries, now in order to retain logs I figured out an absolute solution.
Here, in this blog, the motive is to sync the logs from dying instances at the time of their termination. This will be done using AWS Services, the goal is to trigger a Powershell Script in the instance using SSM which sync logs to S3 Bucket with sufficient information about the dying instances. For this we will require 2 things:

1) Configuring SSM agent to be able to talk to Ec2 Instances
2) Ec2 Instances being able to write to S3 Buckets

For the tutorial we will be using Microsoft Windows Server 2012 R2 Base with the AMI ID: ami-0f7af6e605e2d2db5

A Blueprint of the scenario to be understood below:

1) Configuring SSM agent to be able to talk to Ec2 Instances
 
SSM Agent is installed by default on Windows Server 2016 instances and instances created from Windows Server 2003-2012 R2 AMIs published in November 2016 or later. Windows AMIs published before November 2016 use the EC2Config service to process requests and configure instances.
If your instance is a Windows Server 2003-2012 R2 instance created before November 2016, then EC2Config must be upgraded on the existing instances to use the latest version of EC2Config. By using the latest EC2Config installer, you install SSM Agent side-by-side with EC2Config. This side-by-side version of SSM Agent is compatible with your instances created from earlier Windows AMIs and enables you to use SSM features published after November 2016.
 
This simple script can be used to update Ec2Config and then layer it with the latest version of SSM agent. This will always install AwsCli which is used to push logged archives to S3
 

#ScriptBlock

if(!(Test-Path -Path C:\Scripts )){
mkdir C:\Tmp
}
cd C:/Tmp
wget https://s3.ap-south-1.amazonaws.com/asg-termination-logs/Ec2Install.exe -OutFile Ec2Config.exe
wget https://s3.amazonaws.com/ec2-downloads-windows/SSMAgent/latest/windows_amd64/AmazonSSMAgentSetup.exe -OutFile ssmagent.exe
wget https://s3.amazonaws.com/aws-cli/AWSCLI64PY3.msi -OutFile awscli.msi
wget https://s3.amazonaws.com/aws-cli/AWSCLISetup.exe -OutFile awscli.exe
Invoke-Command -ScriptBlock {C:\Tmp\Ec2Config.exe /Ec /S /v/qn }
sleep 20
Invoke-Command -ScriptBlock {C:\Tmp\awscli.exe /Ec /S /v/qn }
sleep 20
Invoke-Command -ScriptBlock {C:\Tmp\ssmagent.exe /Ec /S /v/qn }
sleep 10
Restart-Service AmazonSSMAgent
Remove-Item C:\Tmp

 
An IAM Role is Required for SSM to Ec2 Instance Conversation:
IAM instance role: Verify that the instance is configured with an AWS Identity and Access Management (IAM) role that enables the instance to communicate with the Systems Manager API.
 
Add instance profile permissions for Systems Manager managed instances to an existing role
  • Open the IAM console at https://console.aws.amazon.com/iam/.
  • In the navigation pane, choose Roles, and then choose the existing role you want to associate with an instance profile for Systems Manager operations.
  • On the Permissions tab, choose Attach policy.
  • On the Attach policy page, select the check box next to AmazonEC2RoleforSSM, and then choose Attach policy.
Now, Navigate to Roles > and select your role. 
That should look like:
 
index
 
2) Ec2 Instances being able to write to S3 Buckets
 
An IAM Role is Required for Ec2 to be able to write to S3:
 
IAM instance role: Verify that the instance is configured with an AWS Identity and Access Management (IAM) role that enables the instance to communicate with the S3 API.
 
Add instance profile permissions for Systems Manager managed instances to an existing role
  • Open the IAM console at https://console.aws.amazon.com/iam/.
  • In the navigation pane, choose Roles, and then choose the existing role you want to associate with an instance profile for Systems Manager operations.
  • On the Permissions tab, choose Attach policy.
  • On the Attach policy page, select the check box next to AmazonS3FullAccess, and then choose Attach policy.
That should look like:
 
index
 

This Powershell script saved in C:/Scripts/termination.ps1 will pick up log files from:

$SourcePathWeb:

and will output logs into:

$DestFileWeb
with a IP and date-stamp to recognize and identify the instances and where the logs originate from later.
Make sure that the s3 bucket name and –region and source of log files is changed according to the preferences.
 

#ScriptBlock

$Date=Get-Date -Format yyyy-MM-dd
$InstanceName=”TerminationEc2″
$LocalIP=curl http://169.254.169.254/latest/meta-data/local-ipv4 -UseBasicParsing

if((Test-Path -Path C:\Users\Administrator\workdir\$InstanceName-$LocalIP-$Date/$Date )){
Remove-Item “C:\Users\Administrator\workdir\$InstanceName-$LocalIP-$Date/$Date” -Force -Recurse
}

New-Item -path “C:\Users\Administrator\workdir\$InstanceName-$LocalIP-$Date/$Date” -type directory
$SourcePathWeb=”C:\Source\Application\web\logs”
$DestFileWeb=”C:\Users\Administrator\workdir\$InstanceName-$LocalIP-$Date/$Date/logs.zip”

Add-Type -assembly “system.io.compression.filesystem”
[io.compression.zipfile]::CreateFromDirectory($SourcePathWeb, $DestFileWeb)

C:\’Program Files’\Amazon\AWSCLI\bin\aws.cmd s3 cp C:\Users\Administrator\workdir s3://terminationec2 –recursive –exclude “*.ok” –include “*” –region us-east-1

If the above settings are done fine then manually the script should produce a success suggesting output:


indexindex

Check your S3, Bucket for seeing if it has synced logs to there. Now, because the focus of this blog trigger a Powershell Script in the instance using SSM which syncs the logs to S3 Bucket so we will try running the script through SSM > Run Command.

Select and run of the instances having the above script and configuration. The output should be pleasing.

index

The AMI used by the ASG should have the above configuration (Can be archived via created a ami from ec2 having above config and then adding it into Launch Configuration of the ASG). The ASG we have here for the tutorial is named after my last name : “group_kaien”.

Now, the last and the Most important step is configuration theCloudwatch > Event > Rules.

Navigating to Cloudwatch>Event>Rules: Create Rule.
 
index
 

This would return the following JSON config:

{
“source”: [
“aws.autoscaling”

],
“detail-type”: [
“EC2 Instance Terminate Successful”,
“EC2 Instance-terminate Lifecycle Action”
],
“detail”: {
“AutoScalingGroupName”: [
“group_kaien”
]
}
}
 
On the right side of Targets:
 
Select
 
SSM Run Command:
  • Document: AwsRunPowerShellScript
  • Target key: “Instanceids or tag:
  • Target Values:
 Configure parameter
  • Commands: .\termination.ps1
  • WorkingDirectory: C:\Scripts.ps1
  • ExecutionTimeout: 3600 is default
Making sure that on termination event happening, the powershell script is run and it syncs logs to S3. This is what our configuration looks like:
 
index
 

For more on setting up Cloudwatch Events refer :
https://docs.aws.amazon.com/AmazonCloudWatch/latest/events/CWE_GettingStarted.html

Wait for the AutoScaling Policies to run such that new instances are created and terminated, with above configuration. The terminating instances will sync their logs S3 before they are fully terminated. Here’s the output on S3 for me after a scale down activity was done.
 
index
 

Conclusion

Now with this above, we have learned how to export logs to S3 automatically from a dying instance, with the correct date/time stamp as mentioned in the termination.ps1 script.
Hence, fulfilling the scope of the blog.
Stay tuned for more

Prometheus Overview and Setup

Overview

Prometheus is an opensource monitoring solution that gathers time series based numerical data. It is a project which was started by Google’s ex-employees at SoundCloud. 

To monitor your services and infra with Prometheus your service needs to expose an endpoint in the form of port or URL. For example:- {{localhost:9090}}. The endpoint is an HTTP interface that exposes the metrics.

For some platforms such as Kubernetes and skyDNS Prometheus act as directly instrumented software that means you don’t have to install any kind of exporters to monitor these platforms. It can directly monitor by Prometheus.

One of the best thing about Prometheus is that it uses a Time Series Database(TSDB) because of that you can use mathematical operations, queries to analyze them. Prometheus uses SQLite as a database but it keeps the monitoring data in volumes.

Pre-requisites

  • A CentOS 7 or Ubuntu VM
  • A non-root sudo user, preferably one named prometheus

Installing Prometheus Server

First, create a new directory to store all the files you download in this tutorial and move to it.

mkdir /opt/prometheus-setup
cd /opt/prometheus-setup
Create a user named “prometheus”

useradd prometheus

Use wget to download the latest build of the Prometheus server and time-series database from GitHub.


wget https://github.com/prometheus/prometheus/releases/download/v2.0.0/prometheus-2.0.0.linux-amd64.tar.gz
The Prometheus monitoring system consists of several components, each of which needs to be installed separately.

Use tar to extract prometheus-2.0.0.linux-amd64.tar.gz:

tar -xvzf ~/opt/prometheus-setup/prometheus-2.0.0.linux-amd64.tar.gz .
 Place your executable file somewhere in your PATH variable, or add them into a path for easy access.

mv prometheus-2.0.0.linux-amd64  prometheus
sudo mv  prometheus/prometheus  /usr/bin/
sudo chown prometheus:prometheus /usr/bin/prometheus
sudo chown -R prometheus:prometheus /opt/prometheus-setup/
mkdir /etc/prometheus
mv prometheus/prometheus.yml /etc/prometheus/
sudo chown -R prometheus:prometheus /etc/prometheus/
prometheus --version
  

You should see the following message on your screen:

  prometheus,       version 2.0.0 (branch: HEAD, revision: 0a74f98628a0463dddc90528220c94de5032d1a0)
  build user:       root@615b82cb36b6
  build date:       20171108-07:11:59
  go version:       go1.9.2
Create a service for Prometheus 

sudo vi /etc/systemd/system/prometheus.service
[Unit]
Description=Prometheus

[Service]
User=prometheus
ExecStart=/usr/bin/prometheus --config.file /etc/prometheus/prometheus.yml --storage.tsdb.path /opt/prometheus-setup/

[Install]
WantedBy=multi-user.target
systemctl daemon-reload

systemctl start prometheus

systemctl enable prometheus

Installing Node Exporter


Prometheus was developed for the purpose of monitoring web services. In order to monitor the metrics of your server, you should install a tool called Node Exporter. Node Exporter, as its name suggests, exports lots of metrics (such as disk I/O statistics, CPU load, memory usage, network statistics, and more) in a format Prometheus understands. Enter the Downloads directory and use wget to download the latest build of Node Exporter which is available on GitHub.

Node exporter is a binary which is written in go which monitors the resources such as cpu, ram and filesystem. 

wget https://github.com/prometheus/node_exporter/releases/download/v0.15.1/node_exporter-0.15.1.linux-amd64.tar.gz

You can now use the tar command to extract : node_exporter-0.15.1.linux-amd64.tar.gz

tar -xvzf node_exporter-0.15.1.linux-amd64.tar.gz .

mv node_exporter-0.15.1.linux-amd64 node-exporter

Perform this action:-

mv node-exporter/node_exporter /usr/bin/

Running Node Exporter as a Service

Create a user named “prometheus” on the machine on which you are going to create node exporter service.

useradd prometheus

To make it easy to start and stop the Node Exporter, let us now convert it into a service. Use vi or any other text editor to create a unit configuration file called node_exporter.service.


sudo vi /etc/systemd/system/node_exporter.service
This file should contain the path of the node_exporter executable, and also specify which user should run the executable. Accordingly, add the following code:

[Unit]
Description=Node Exporter

[Service]
User=prometheus
ExecStart=/usr/bin/node_exporter

[Install]
WantedBy=default.target

Save the file and exit the text editor. Reload systemd so that it reads the configuration file you just created.


sudo systemctl daemon-reload
At this point, Node Exporter is available as a service which can be managed using the systemctl command. Enable it so that it starts automatically at boot time.

sudo systemctl enable node_exporter.service
You can now either reboot your server or use the following command to start the service manually:
sudo systemctl start node_exporter.service
Once it starts, use a browser to view Node Exporter’s web interface, which is available at http://your_server_ip:9100/metrics. You should see a page with a lot of text:

Starting Prometheus Server with a new node

Before you start Prometheus, you must first edit a configuration file for it called prometheus.yml.

vim /etc/prometheus/prometheus.yml
Copy the following code into the file.

# my global configuration which means it will applicable for all jobs in file
global:
  scrape_interval:     15s # Set the scrape interval to every 15 seconds. Default is every 1 minute. scrape_interval should be provided for scraping data from exporters 
  evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute. Evaluation interval checks at particular time is there any update on alerting rules or not.

# Load rules once and periodically evaluate them according to the global 'evaluation_interval'. Here we will define our rules file path 
#rule_files:
#  - "node_rules.yml"
#  - "db_rules.yml"

# A scrape configuration containing exactly one endpoint to scrape: In the scrape config we can define our job definitions
scrape_configs:
  # The job name is added as a label `job=` to any timeseries scraped from this config.
  - job_name: 'node-exporter'
    # metrics_path defaults to '/metrics'
    # scheme defaults to 'http'. 
    # target are the machine on which exporter are running and exposing data at particular port.
    static_configs:
      - targets: ['localhost:9100']
After adding configuration in prometheus.yml. We should restart the service by

systemctl restart prometheus
This creates a scrape_configs section and defines a job called a node. It includes the URL of your Node Exporter’s web interface in its array of targets. The scrape_interval is set to 15 seconds so that Prometheus scrapes the metrics once every fifteen seconds. You could name your job anything you want, but calling it “node” allows you to use the default console templates of Node Exporter.
Use a browser to visit Prometheus’s homepage available at http://your_server_ip:9090. You’ll see the following homepage. Visit http://your_server_ip:9090/consoles/node.html to access the Node Console and click on your server, localhost:9100, to view its metrics.

Logstash Timestamp

Introduction

A few days back I encountered with a simple but painful issue. I am using ELK to parse my application logs  and generate some meaningful views. Here I met with an issue which is, logstash inserts my logs into elasticsearch as per the current timestamp, instead of the actual time of log generation.
This creates a mess to generate graphs with correct time value on Kibana.
So I had a dig around this and found a way to overcome this concern. I made some changes in my logstash configuration to replace default time-stamp of logstash with the actual timestamp of my logs.

Logstash Filter

Add following piece of code in your  filter plugin section of logstash’s configuration file, and it will make logstash to insert logs into elasticsearch with the actual timestamp of your logs, besides the timestamp of logstash (current timestamp).
 
date {
  locale => "en"
  timezone => "GMT"
  match => [ "timestamp", "yyyy-mm-dd HH:mm:ss +0000" ]
}
In my case, the timezone was GMT  for my logs. You need to change these entries  “yyyy-mm-dd HH:mm:ss +0000”  with the corresponding to the regex for actual timestamp of your logs.

Description

Date plugin will override the logstash’s timestamp with the timestamp of your logs. Now you can easily adjust timezone in kibana and it will show your logs on correct time.
(Note: Kibana adjust UTC time with you bowser’s timezone)

Classless Inter Domain Routing Made Easy (Cont..)

Introduction :

As we had a discussion  about Ip addresses and their classes in the previous blog,we can now start with Sub-netting.

Network Mask /Subnet Mask –

As mask means to cover something,
IP Address is made up of two components, One is the network address and the other is the host address.The Ip Address needs to be separated into the network and host address, and this separation of network and host address in done by Subnet Mask.The host part of an IP Address is further divided into subnet and host address if more subnetworks are needed and this can be done by subnetting. It is called as a subnet mask or Network mask as it is used to identify network address of an IP address by performing a bitwise AND operation on the netmask.
Subnet Mask is of 32 Bit and is used to divide the network address and host addresses of an IP.
In a Subnet Mask all the network bits are set to 1’s and all the host bits are set to 0’s.
 
Whenever we see an IP Address – We can easily Identify that
WHAT IS NETWORK PART OF THAT IP
WHAT IS THE HOST PART OF THAT IP
 
FORMAT :
mmmmmmmm.mmmmmmmm.mmmmmmmm.mmmmmmmm
(Either it will have 1 or 0 Continuously)
EXAMPLE :
A Class Network Mask
In Binary : 11111111.00000000.00000000.00000000         – First 8 Bits will be Fixed
In Decimal : 255.0.0.0
Let the IP Given is – 10.10.10.10
When we try to Identify it we know that it belong to class A, So the subnet mask will be : 255.0.0.0
And the Network Address will be : 10.0.0.0
 
B Class Network Mask  
In Binary : 11111111.11111111.00000000.00000000           – First 16 Bits will be Fixed
In Decimal : 255.255.0.0
Let the IP Given is -150.150.150.150
When we try to Identify it we know that it belong to class B, So the subnet mask will be : 255.255.0.0
And the Network Address will be : 150.150.0.0
 
C Class Network Mask  
In Binary : 11111111.111111111.11111111.00000000           – First 32 Bits will be Fixed
In Decimal : 255.255.255.0
Let the IP Given is – 200.10.10.10
When we try to Identify it we know that it belong to class C, So the subnet mask will be : 255.255.255.0
And the Network Address will be : 200.10.10.0

Subnetting :

The method of dividing a network into two or more networks is called subnetting.
A subnetwork, or subnet, is a logically subdivision of an IP network
Subnetting provides Better Security
Smaller collision and Broadcast  Domains
Greater administrative control of each network.
Subnetting – WHY ??
Answer : Shortage of IP Addresses
SOLUTIONS : –
1) SUBNETTING – To divide Bigger network into the smaller networks and to reduce the wastage
2) NAT –  Network Address Translation
3) Classless IP Addressing –
No Bits are reserved for Network and Host
 
**Now the Problem that came is how to Identify the Class of IP Address :**
Let a IP Be : 10.10.10.10
If we talk about classful we can say it is of class A But in classless : We can check it through subnetwork mask.
255.255.255.0
So by this we can say that first 24 bits are masked for network and the left 8 are for host.
Bits Borrowed from Host and added to Network
Network ID(N)
Network ID(N)
Host ID(H)
Host ID(H)
Network ID(N)
Network ID(N)
Subnet
Host ID(H)
Network ID(N)
Network ID(N)
Subnet
Subnet/Host
Let we have a
150.150.0.0 – Class Identifier/Network Address
150.150.2.4 – Host Address – IP GIVEN TO A HOST
255.255.255.0 – Subnet Mask
150.150.2.0 – Subnet Address

CIDR : Classless Inter Domain Routing

CIDR (Classless Inter-Domain Routing, sometimes called supernetting) is a way to allow more flexible allocation of Internet Protocol addresses than was possible with the original system of IP Address classes. As a result, the number of available Internet addresses was greatly increased, which along with widespread use of network address translation, has significantly extended the useful life of IPv4.
Let a IP be – 200.200.200.200
 
Network ID(N)
Host ID(H)
——–24 Bit ——– ——-8 bit ———–
   
Network Mask tells that the number of 1’s are Masked
Here First 24 Bits are Masked
In Decimal : 255.255.255.0
In Binary : 11111111.11111111.11111111.00000000
   Here the total Number of 1’s : 24
So we can say that 24 Bits are masked.
 
This method of Writing the network mask can be represented in one more way
And that representation is called as CIDR METHOD/CIDR NOTATION

CIDR  – 200.200.200.200/24
24 : Is the Number of Ones – Or we can say Bits Masked
Basically the method ISP’s(Internet Service Provider)use to  allocate an amount of addresses to a company, a home
 
EX :
190.10.20.30/28 : Here 28 Bits are Masked that represents the Network and the remaining 4 bits represent the Host
/ – Represents how many bits are turned on (1s)

CLASS C SUBNETTING :

 
Determining Available Host Address :
 
200
10
20
0
11001000               00001010               00010100                 00000000 – 1
                                                                                              00000001 – 2     
                                      00000011 – 3
                                                                          .
                                                                                                    .
                                                                                                    .
                                                                                              11111101 – 254
                                                                                              11111110 – 255
                                                                                              11111111 – 256     
                                                                                                                    -2
                                                                                                               ———
                                                                                                                   254
    2^N – 2  = 2^8 -2 = 254
           (Coz we have 8 bits in this case)               – 2 (Because 2 Address are Reserved)
254 Address are available here
 
FORMULAS :
 
Number of Subnets : ( 2^x ) – 2     (x : Number of Bits Borrowed)
Number of Hosts : ( 2^y ) – 2         (y : Number of Zero’s)
Magic Number or Block Size = Total Number of Address : 256 – Mask
Let a IP ADDRESS BE 200.10.20.20/24
Number of subnets : 5
 
Network Address   :
200
10
20
20
255
255
255
0
(as total Number of 1’s : 24)
IP in Binary
11001000
00001010
00010100
00010100
MASK
11111111
11111111
11111111
00000000

And Operation in IP And Mask
11001000
00001010
00010100
00000000
In Binary
200
10
20
0
As we need 5 Subnets :
2^n -2 => 5
So the value of n = 3 that satisfies the condition
So, We need to turn 3 Zero’s to One’s to create 5 subnets
 
200
10
20
0
11001000
00001010
00010100
00000000
 
11001000
00001010
00010100
11100000
 (3 Zero’s changed to 3 one’s)    
200
10
20
224
                                                                                  
Subnet 0   
200
10
20
0/27  
Subnet 1                                           +32 – Block Size
200
10
20
32/27
Subnet 2                                            +32
200
10
20
64/27
Subnet 3
200
10
20
96/27
Subnet 4
200
10
20
128/27
Subnet 5   
200
10
20
160/27
Subnet 6
200
10
20
192/27
Subnet 7
200
10
20
224/27

How to Put Host ADD.
Subnet 0   
200
10
20
0/27  
Subnet Broadcast Number 0
200
10
20
31 /27  
Subnet 1                                           +32 – Block Size
200
10
20
31/27
200
10
20
32/27
200
10
20
33/27
                                                          .
                                                          .
                                                          .
200
10
20
62/27
Subnet Broadcast Subnet 1
200
10
20
63/27
200.10.20.33 ….and so on till 200.10.20.62   – 13 Host can be assigned IP Address.

Conclusion :

As the world is growing rapidly towards digitalization, use of IP Addresses is also increasing, So to decrease the wastage of IP Addresses, the implementation of CIDR is important that allows more organizations and users to take advantage of IPV4.

Classless Inter Domain Routing Made Easy

Introduction :
One day I was working with VPC (Virtual Private Cloud) inside AWS(Amazon Web Services), where I had a need to calculate the CIDR notation of an IP address and subnet combinations.
I had to use online tools to calculate the Subnets and CIDR every time when I was working with VPC, but I found it interesting that how the network get  broken into different small Networks. So, finally I decided why not to learn CIDR Methods, and then calculate it by my own side instead of using tools every time.
But the questions that striked in my mind were:
  • What is CIDR ?
  • How CIDR Came into Picture ?
  • What CIDR do ?
For Understanding CIDR – (Classless Inter-Domain Routing) few thing need to be cleared before :

1. IP Addresses
2. Structure of IP Address
3. Internet Protocol Address Types
4. Classes
5. Network Mask
6. Subnetting

IP Address –

It is the Address of the Computer, Laptop, Printers or even of the Mobile Sets.
Everyone has some Address, so as these devices also have an Internet Protocol Address (IP Address), also called as Logical Address.
In a Network there are many Computers …
Network..??
A Network is a group of two or more Computers Linked Together.
So When there are Many Computers in a Network, We need to uniquely identify each Computer, so there IP ADDRESS works as an Unique Identifier for Computers and Other Devices.
For Example : There are Twin Sisters, How we are going to Identify them differently  
By their Name that are unique for each of them.
Here Name of the Girls are the IP Addresses that will be unique and the two Girls are the two Devices.

Structure of IP Address –

Now the Question is How do an IP Address looks like??
IP ADDRESS : 192.168.33.10
IP ADDRESS is made up of 32-Bit – 8.8.8.8 = (8+8+8+8=32 Bits)
A bit (short for binary digit) is the smallest unit of data in a computer.
Binary Conversion for 192 :
192 :    128      64       32      16           8      4          2         1
              1        1         0         0           0       0          0         0
          Bit 1    Bit 2    Bit 3   Bit 4     Bit 5    Bit 6   Bit 7     Bit 8    – Total Bit = 8
128+64 = 192
So, 0’s for Other and 1 for the Number whose sum will be 192
Binary Conversion for 168 :
168 :    128      64       32         16       8           4         2          1
             1          0         1           0        1           0         0          0
           Bit 1    Bit 2    Bit 3   Bit 4     Bit 5    Bit 6   Bit 7     Bit 8 – Total Bit = 8
Binary Conversion for 33 :
  33 :   128        64       32        16       8           4         2          1
             0          0         1           0        0           0         0          1
           Bit 1    Bit 2    Bit 3   Bit 4     Bit 5    Bit 6   Bit 7     Bit 8 – Total Bit = 8
Binary Conversion for 10 :
10 :     128       64       32         16       8          4          2          1
             0          0         0           0        1           0         1          0
           Bit 1    Bit 2    Bit 3   Bit 4     Bit 5    Bit 6   Bit 7     Bit 8 – Total Bit = 8
8.8.8.8 – total of 32 Bit.
Dotted Decimal Notation : In dot form 4 Sections are called as OCTETS – Vendor Neutral Term for Bytes.
Let a IP Be : 200.10.20.30
Inside a Network : 200.10.20 – will remain same and 30 will be unique for each.

Type of IP Address –

  1. Assignment Method
  2. Classes : 1) Classful
                    2) Classless
  3. Public / Private
  4. Version

Assignment Methods :

Assignment Method is method that defines how to assign an IP address to a Device.
IP Address can be assigned in two ways
1) Static IP Address
Static IP Address is the IP Address in which configuration is done Manually and is used in small networks.
2) Dynamic IP Address
Dynamic IP Address is the IP Address in which the configuration is done by the Computer Interface or by the Host Interface – DHCP (Dynamic Host Configuration Protocol)
— Configuration is Automatic–

Classes :

classes define that in an IP, How much part will be for Network and How much is for Host.
There are 2 types of classes in IP Addressing :
  1. Classful
  2. Classless
CLASSFUL : IP Address are divided into 5 Classes;
Class A : 0 – 126                         N.H.H.H              Assigned for Large Organization
127                                               N.H.H.H             Assigned for the Loopback
Class B : 128 – 191                     N.N.H.H              Assigned for Medium Companies
Class C : 192 – 223                     N.N.N.H              Assigned for Small Organizations
Class D : 224 – 239                                                 Assigned for Multicasting
Class E : 240 – 255                                                 Assigned for Experimental Purpose

CLASSLESS : Classless addressing is an  IP address where a subnet mask does not define its class.  Subnet mask can be anywhere between bit 0 and bit 31.
CLASS A IP ADDRESS :
Range of Class A IP Address :  0.0.0.0 – 127.255.255.255
Network ID : 8 Bit
Host ID : 24 Bit (8+8+8)
  • IP Address begins with 0,First Bit will always be Zero
  • 7 Remaining Bits in Network part : Only 128 Possible class A Network
  • 24 Bits in Local Part : Over 16 million hosts per Class A Network
  • All class A network parts are assigned or reserved.
Network ID(N)
Host ID(H)
Host ID(H)
Host ID(H)
0                     7 8                                                                31
0NNNNNNN       .      HHHHHHHH     .      HHHHHHHH   .         HHHHHHHH
In Binary :
Class A starts from : 00000000.00000000.00000000.00000000
Class A ends at      : 01111111.11111111.11111111.11111111
In Decimal :
Class A IP Address is from 0.0.0.0 to 127.255.255.255
Number of Networks : 2^7 = 128
Number of Hosts : 2^24
SOME EXCEPTIONS IN CLASS A : Cannot be assigned to host
0.0.0.0 : For Self check – Represent Default Network or M
0.255.255.255 : For Self check – Represent Default Network or My IP
127.0.0.0 : Loop Back Address Range : solve NIC Problem
127.255.255.255 : Loop Back Address Range : solve NIC Problem

CLASS B IP ADDRESS :
Range of Class B IP Address : 128.0.0.0 – 191.255.255.255
Network ID : 16 Bit(8+8)
Host ID : 16 Bit (8+8)
  • First two Bit will always be One and Zero
  • 14 Bits in Network part – Over 16,000 possible Class B Network
  • 16 Bits in Local Part  – Over 65,000 possible Hosts
Network ID(N)
Network ID(N)
Host ID(H)
Host ID(H)
0                                        15 | 16                                                    31
10NNNNNN          .     NNNNNNNN     . HHHHHHHH       . HHHHHHHH
In Binary :
Class B starts fr0m : 10000000.00000000.00000000.00000000
Class B ends at        : 10111111.11111111.11111111.11111111
In Decimal :
Class B IP Address is from  128.0.0.0 to 191.255.255.255
Number of Networks : 2^14
Number of Hosts : 2^16
SOME EXCEPTIONS IN CLASS B : Cannot be assigned to host
169.254.X.X : Reserved for APIPA (Automatic Private IP Address) – Host take IP Automatically ifit doesn’t get any DHCP Server in the Network.

CLASS C IP ADDRESS :
Range of Class B IP Address : 192.0.0.0 – 223.255.255.255
Network ID : 24 Bit(8+8+8)
Host ID : 8 Bit (8)
**Most Popular and Commonly Used**
  • First three Bit will always be One,One and Zero
  • 21 Bits in Network part – Over 2 Million  possible Class C Network
  • 8 Bits in Local Part  – Only  256 possible Hosts per class C Network
Network ID(N)
Network ID(N)
Network ID(N)
Host ID(H)
0                                                                        23 | 24                             31
110NNNNN            .    NNNNNNNN   .      NNNNNNNN     .      HHHHHHHH
In Binary :
Class C starts from : 1100000.00000000.00000000.00000000
Class C ends at        : 11011111.11111111.11111111.11111111
In Decimal :
Class C IP Address is from  192.0.0.0 to 223.255.255.255
Number of Networks : 2^21
Number of Hosts : 2^8


CLASS D IP ADDRESS :
Range : 224.0.0.0 – 239.255.255.255
IP Address begins with 1110

Used for Multicasting, Not defining networks.
  • Sending messages to group of hosts
  • just to one (Unicasting)
  • ALL HOSTS (Broadcasting)
  • Say to send a videoconference stream to a group of receivers
In Binary :
Class D starts from : 11100000.00000000.00000000.00000000
Class D end at        : 11101111.11111111.11111111.11111111
In Decimal :
Class D IP Address is from  224.0.0.0 to 239.255.255.255
224.0.0.5 – OSPF
All OSPF Routers address is used to send HELLO PACKETS
224.0.0.6 – OSPF
All the routers address is used to send OSPF routing information to designated routers on a network segment.
224.0.0.9 – The Routing Information Protocol (RIP) version 2 group address is used to send routing information to all RIP2-aware routers on a network segment.
224.0.0.10 – EIGRP
used to send routing information to all EIGRP routers on a network segment.
224.0.0.18 – Virtual Router Redundancy Protocol.

Private/Public:

PUBLIC :
A public also called as  External IP address is the one that your ISP (Internet Service Provider) provides to identify your home network to the outside world. It is an IP address that is unique throughout the entire Internet.
When you’re setting up your router, if your ISP issued you a static IP address, you enter it into your router’s settings. For a dynamic IP address, you specify DHCP in your router’s network settings. DHCP is Dynamic Host Control Protocol. It tells your router to accept whatever public IP address your ISP issues.
Those who wanted not to connect through internet but they wanted to run their network on TCP/IP Protocol
Here came the concept of PRIVATE  IP
PRIVATE :
Just as your network’s public IP address is issued by your ISP, your router issues private (or internal) IP addresses to each network device inside your network. This provides unique identification for devices that are within your home network, such as your computer, your Slingbox, and so on.
THEY ARE NOT ROUTABLE
CLASS A PRIVATE ADDRESS   10.0.0.0 – 10.255.255.255
CLASS B PRIVATE ADDRESS   172.16.0.0 – 172.31.255.255
CLASS C PRIVATE ADDRESS   192.168.0.0 – 192.168.255.255
Internet Protocol Address :
           Reserved IP Address :
  1. Addresses beginning with 127 are reserved for loopback and internal testing – Used for Self Testing that TCP/IP is properly working or not.
  2. XXX.0.0.0 reserved for Network Address
  3. XXX.255.255.255 reserved for Broadcast
  4. 0.0.0.0 – First Address – Represent Local Network / Used for Default Routing
  5. 255.255.255.255 – Broadcast
Example : Let a Class A IP Address be – 101.101.101.101
               Network Address – 101.0.0.0
               BroadCast Address – 101.255.255.255
 : Let a Class B IP Address be – 150.150.150.150
               Network Address – 150.150.0.0
               BroadCast Address – 150.150.255.255

I hope that gives you a good knowledge of IP Addresses and their classes.
Now, We can move on to what sub-netting is, in my next blog.
Please Follow this link to get on to sub-netting –
Classless Inter Domain Routing Made Easy (Cont..)

Snoopy + ELK : Exhibit sudo commands in Kibana Dashboard

Logging User Commands: Snoopy Logger

About Snoopy Logger

Snoopy logs all the commands that are ran by any user to a log file. This is helpful for auditing and keep an eye on user activities.

Automated Installation

For Automated Installation/Configuration of Snoopy we have created a Puppet module and Ansible Role.

Manual Installation

To install the latest STABLE version of Snoopy, use these commands:
rm -f snoopy-install.sh
wget -O snoopy-install.sh https://github.com/a2o/snoopy/raw/install/doc/install/bin/snoopy-install.sh
chmod 755 snoopy-install.sh
./snoopy-install.sh stable

Output

This is what typical Snoopy output looks like:
2015-02-11T19:05:10+00:00 labrat-1 snoopy[896]: [uid:0 sid:11679 tty:/dev/pts/2 cwd:/root filename:/usr/bin/cat]: cat /etc/fstab.BAK
2015-02-11T19:05:15+00:00 labrat-1 snoopy[896]: [uid:0 sid:11679 tty:/dev/pts/2 cwd:/root filename:/usr/bin/rm]: rm -f /etc/fstab.BAK
These are default output locations on various Linux distributions:

  • CentOS: /var/log/secure
  • Debian: /var/log/auth.log
  • Ubuntu: /var/log/auth.log
  • others: /var/log/messages (potentially, not necessarily)

For actual output destination check your syslog configuration.
Snoopy provides a configuration file “/etc/snoopy.ini” where you can configure snoopy to generate logs. By default snoopy logs only uid, but doesn’t logs username in logs, so we have to change configuration to get username in logs.You may also specify the log path where you want to generate the snoopy logs.
For getting username in logs edit “/etc/snoopy.ini” and under [snoopy] section add the following line:
message_format = “[username:%{username} uid:%{uid} sid:%{sid} tty:%{tty} cwd:%{cwd} filename:%{filename}]: %{cmdline}”
The output of logs is  shown below:
Feb 25 07:47:27 vagrant-ubuntu-trusty-64 snoopy[3163]: [username:root uid:0 sid:1828 tty:/dev/pts/0 cwd:/root filename:/usr/bin/vim]: vim /etc/snoopy.ini

Enable/Disable Snoopy

To enable snoopy, issue the following command:
snoopy-enable
To disable snoopy, issue the following command:
snoopy-disable

Using ELK to parse logs

Now that we have logs with suitable information we will write a grok pattern in logstash to parse these logs and generate required fields.
A sample grok pattern will be like this:


filter {

 if [type] == “snoopy” {
   grok {
     match => { “message” => “%{SYSLOGTIMESTAMP:date} %{HOSTNAME:hostname} %{WORD:logger}\[%{INT}\]\: \[%{WORD}\:%{USERNAME:username} %{DATA} %{DATA} %{DATA} %{WORD}\:%{DATA:cwd} %{DATA}\]\: %{GREEDYDATA:exe_command}” }
   }
 if “_grokparsefailure” in [tags] {
   drop { }
 }
 }
}

Here we are generating these fields:
date: Timestamp at which log is generated
hostname: Name of host
logger: Name of logger which is generating logs in our case “snoopy”.
username: Name of user issuing the command
cwd: Absolute path of directory from where the command is executed
exe_command: Command that is executed by user with complete options

Place the above grok pattern in filter section of logstash configuration file which is at “/etc/logstash/conf.d/logstash.conf”. Also include logs from “/var/log/auth.log” to be shipped to logstash server from logstash agent at the client.

Creating Dashboard in Kibana

After that you can see these logs in kibana in “Discover” tab as shown in screenshot:

elkdiscover.png

In the left sidebar you can see all the fields via which you can filter including the fields we set in our grok pattern.Now in the search bar you can search according to specific field and its value. For example to search logs for vagrant user and all sudo commands executed by it, you will write the following query in search bar:
username:vagrant AND exe_command:sudo*
Then from the left sidebar add the fields you want to see, for example add “username”, “exe_command” and “cwd”, which will result to a table as shown below:

elktableselectedfields.png

Now save this search from the icon that is just adjacent to left bar with a suitable name. Then go to “Dashboard” menu and click on “plus” icon to add a dashboard. A screen will appear as shown:

adddashboard.png

Click on “Searches” tab and find your saved search and click over it. A resulting screen will appear which will be added to your dashboard as shown below:

dashboardadded.png

Here you can view tabular data for the sudo commands executed by vagrant user. Similarly you can add more searches by clicking on “plus icon” and add it to the same dashboard.Now save this dashboard by clicking on the “save” icon adjacent to search bar with a suitable name.After that you can easily load this dashboard by clicking on “load” icon adjacent to search bar.

Snoopy : Get, Set and log

Recently we got a requirement to log each and every command executed by the users. Upon googling we found a tool that is  exclusively contrived to accomplish this ambition.

This tool was “Snoopy”. This is a open source project whose microscopic documentation is available at their github page:

 

You can find all installation and configuration doc at their github README.

Why snoopy?

We choose snoopy because it is a very lightweight  tool that is just built with the intention  of logging each and every command executed by user. It consists of only a tiny library which does all the logging.

Use cases:

1). Log each executed command by any user with its arguments
2). Configure the pattern of logs according to requirement using a simple configuration file
3). The logs generated by snoopy can be sent to a central logging server and then can be parsed to get useful information. For example: To get all sudo commands executed by a user.

About Snoopy:

Snoopy logger is a majestic utility which makes the admin work more effortless by providing a log of commands with its arguments executed via shell by any user. It comes with a configuration file “/etc/snoopy.ini” where you can configure how the logs are generated and the information in the logs you want to have.

Automating Snoopy                            installation/configuration:

We will make the procedure easy for installation and configuration of snoopy by creating puppet module and ansible role. We’ll soon be launching a puppet module and an ansible role for it.

 

Also we’ll showcase a useful example of snoopy logger with centralized logging using ELK.