Logstash Timestamp

Introduction

A few days back I encountered with a simple but painful issue. I am using ELK to parse my application logs and generate some meaningful views. Here I met with an issue which is, logstash inserts my logs into elasticsearch as per the current timestamp, instead of the actual time of log generation.

This creates a mess to generate graphs with correct time value on Kibana.

So I had a dig around this and found a way to overcome this concern. I made some changes in my logstash configuration to replace default time-stamp of logstash with the actual timestamp of my logs.

Logstash Filter

Add following piece of code in your filter plugin section of logstash’s configuration file, and it will make logstash to insert logs into elasticsearch with the actual timestamp of your logs, besides the timestamp of logstash (current timestamp).

date {
  locale => "en"
  timezone => "GMT"
  match => [ "timestamp", "yyyy-mm-dd HH:mm:ss +0000" ]
}

In my case, the timezone was GMT for my logs. You need to change these entries “yyyy-mm-dd HH:mm:ss +0000” with the corresponding to the regex for actual timestamp of your logs.

Description

Date plugin will override the logstash’s timestamp with the timestamp of your logs. Now you can easily adjust timezone in kibana and it will show your logs on correct time.

(Note: Kibana adjust UTC time with you bowser’s timezone)

Classless Inter Domain Routing Made Easy (Cont..)

Introduction :

As we had a discussion about Ip addresses and their classes in the previous blog,we can now start with Sub-netting.

Network Mask /Subnet Mask –

As mask means to cover something,

IP Address is made up of two components, One is the network address and the other is the host address.The Ip Address needs to be separated into the network and host address, and this separation of network and host address in done by Subnet Mask.The host part of an IP Address is further divided into subnet and host address if more subnetworks are needed and this can be done by subnetting. It is called as a subnet mask or Network mask as it is used to identify network address of an IP address by performing a bitwise AND operation on the netmask.

Subnet Mask is of 32 Bit and is used to divide the network address and host addresses of an IP.

In a Subnet Mask all the network bits are set to 1’s and all the host bits are set to 0’s.

Whenever we see an IP Address – We can easily Identify that

WHAT IS NETWORK PART OF THAT IP

WHAT IS THE HOST PART OF THAT IP

FORMAT :

mmmmmmmm.mmmmmmmm.mmmmmmmm.mmmmmmmm

(Either it will have 1 or 0 Continuously)

EXAMPLE :

A Class Network Mask

In Binary : 11111111.00000000.00000000.00000000 – First 8 Bits will be Fixed
In Decimal : 255.0.0.0

Let the IP Given is – 10.10.10.10

When we try to Identify it we know that it belong to class A, So the subnet mask will be : 255.0.0.0

And the Network Address will be : 10.0.0.0

B Class Network Mask

In Binary : 11111111.11111111.00000000.00000000 – First 16 Bits will be Fixed

In Decimal : 255.255.0.0

Let the IP Given is -150.150.150.150

When we try to Identify it we know that it belong to class B, So the subnet mask will be : 255.255.0.0

And the Network Address will be : 150.150.0.0

C Class Network Mask

In Binary : 11111111.111111111.11111111.00000000 – First 32 Bits will be Fixed

In Decimal : 255.255.255.0

Let the IP Given is – 200.10.10.10

When we try to Identify it we know that it belong to class C, So the subnet mask will be : 255.255.255.0

And the Network Address will be : 200.10.10.0

Subnetting :

The method of dividing a network into two or more networks is called subnetting.

A subnetwork, or subnet, is a logically subdivision of an IP network

Subnetting provides Better Security

Smaller collision and Broadcast Domains

Greater administrative control of each network.

Subnetting – WHY ??

Answer : Shortage of IP Addresses

SOLUTIONS : –

1) SUBNETTING – To divide Bigger network into the smaller networks and to reduce the wastage

2) NAT – Network Address Translation

3) Classless IP Addressing –

No Bits are reserved for Network and Host

**Now the Problem that came is how to Identify the Class of IP Address :**

Let a IP Be : 10.10.10.10

If we talk about classful we can say it is of class A But in classless : We can check it through subnetwork mask.

255.255.255.0

So by this we can say that first 24 bits are masked for network and the left 8 are for host.

Bits Borrowed from Host and added to Network

Network ID(N)

Host ID(H)

Network ID(N)

Subnet

Host ID(H)

Network ID(N)

Subnet

Subnet/Host

Let we have a

150.150.0.0 – Class Identifier/Network Address

150.150.2.4 – Host Address – IP GIVEN TO A HOST

255.255.255.0 – Subnet Mask

150.150.2.0 – Subnet Address

CIDR : Classless Inter Domain Routing

CIDR (Classless Inter-Domain Routing, sometimes called supernetting) is a way to allow more flexible allocation of Internet Protocol addresses than was possible with the original system of IP Address classes. As a result, the number of available Internet addresses was greatly increased, which along with widespread use of network address translation, has significantly extended the useful life of IPv4.

Let a IP be – 200.200.200.200

Network ID(N)

Host ID(H)

——–24 Bit ——– ——-8 bit ———–

Network Mask tells that the number of 1’s are Masked

Here First 24 Bits are Masked

In Decimal : 255.255.255.0

In Binary : 11111111.11111111.11111111.00000000

Here the total Number of 1’s : 24

So we can say that 24 Bits are masked.

This method of Writing the network mask can be represented in one more way

And that representation is called as CIDR METHOD/CIDR NOTATION

CIDR – 200.200.200.200/24

24 : Is the Number of Ones – Or we can say Bits Masked

Basically the method ISP’s(Internet Service Provider)use to allocate an amount of addresses to a company, a home

EX :

190.10.20.30/28 : Here 28 Bits are Masked that represents the Network and the remaining 4 bits represent the Host

/ – Represents how many bits are turned on (1s)

CLASS C SUBNETTING :

Determining Available Host Address :

200

11001000 00001010 00010100 00000000 – 1

00000001 – 2

00000011 – 3

11111101 – 254

11111110 – 255

11111111 – 256

-2

———

254

2^N – 2 = 2^8 -2 = 254

(Coz we have 8 bits in this case) – 2 (Because 2 Address are Reserved)

254 Address are available here

FORMULAS :

Number of Subnets : ( 2^x ) – 2 (x : Number of Bits Borrowed)

Number of Hosts : ( 2^y ) – 2 (y : Number of Zero’s)

Magic Number or Block Size = Total Number of Address : 256 – Mask

Let a IP ADDRESS BE 200.10.20.20/24

Number of subnets : 5

Network Address :

200

255

(as total Number of 1’s : 24)

IP in Binary

11001000

00001010

00010100

MASK

11111111

00000000

And Operation in IP And Mask

11001000

00001010

00010100

00000000

In Binary

200

As we need 5 Subnets :

2^n -2 => 5

So the value of n = 3 that satisfies the condition

So, We need to turn 3 Zero’s to One’s to create 5 subnets

200

11001000

00001010

00010100

00000000

11001000

00001010

00010100

11100000

(3 Zero’s changed to 3 one’s)

200

224

Subnet 0

200

0/27

Subnet 1 +32 – Block Size

200

32/27

Subnet 2 +32

200

64/27

Subnet 3

200

96/27

Subnet 4

200

128/27

Subnet 5

200

160/27

Subnet 6

200

192/27

Subnet 7

200

224/27

How to Put Host ADD.

Subnet 0

200

0/27

Subnet Broadcast Number 0

200

31 /27

Subnet 1 +32 – Block Size

200

31/27

200

32/27

200

33/27

200

62/27

Subnet Broadcast Subnet 1

200

63/27

200.10.20.33 ….and so on till 200.10.20.62 – 13 Host can be assigned IP Address.

Conclusion :

As the world is growing rapidly towards digitalization, use of IP Addresses is also increasing, So to decrease the wastage of IP Addresses, the implementation of CIDR is important that allows more organizations and users to take advantage of IPV4.

Classless Inter Domain Routing Made Easy

Introduction :

One day I was working with VPC (Virtual Private Cloud) inside AWS(Amazon Web Services), where I had a need to calculate the CIDR notation of an IP address and subnet combinations.

I had to use online tools to calculate the Subnets and CIDR every time when I was working with VPC, but I found it interesting that how the network get broken into different small Networks. So, finally I decided why not to learn CIDR Methods, and then calculate it by my own side instead of using tools every time.

But the questions that striked in my mind were:

What is CIDR ?
How CIDR Came into Picture ?
What CIDR do ?

For Understanding CIDR – (Classless Inter-Domain Routing) few thing need to be cleared before :

1. IP Addresses

2. Structure of IP Address

3. Internet Protocol Address Types

4. Classes

5. Network Mask

6. Subnetting

IP Address –

It is the Address of the Computer, Laptop, Printers or even of the Mobile Sets.
Everyone has some Address, so as these devices also have an Internet Protocol Address (IP Address), also called as Logical Address.

In a Network there are many Computers …

Network..??

A Network is a group of two or more Computers Linked Together.

So When there are Many Computers in a Network, We need to uniquely identify each Computer, so there IP ADDRESS works as an Unique Identifier for Computers and Other Devices.

For Example : There are Twin Sisters, How we are going to Identify them differently
By their Name that are unique for each of them.

Here Name of the Girls are the IP Addresses that will be unique and the two Girls are the two Devices.

Structure of IP Address –

Now the Question is How do an IP Address looks like??

IP ADDRESS : 192.168.33.10

IP ADDRESS is made up of 32-Bit – 8.8.8.8 = (8+8+8+8=32 Bits)

A bit (short for binary digit) is the smallest unit of data in a computer.

Binary Conversion for 192 :

192 : 128 64 32 16 8 4 2 1

1 1 0 0 0 0 0 0

Bit 1 Bit 2 Bit 3 Bit 4 Bit 5 Bit 6 Bit 7 Bit 8 – Total Bit = 8

128+64 = 192
So, 0’s for Other and 1 for the Number whose sum will be 192

Binary Conversion for 168 :

168 : 128 64 32 16 8 4 2 1

1 0 1 0 1 0 0 0

Bit 1 Bit 2 Bit 3 Bit 4 Bit 5 Bit 6 Bit 7 Bit 8 – Total Bit = 8

Binary Conversion for 33 :

33 : 128 64 32 16 8 4 2 1

0 0 1 0 0 0 0 1

Bit 1 Bit 2 Bit 3 Bit 4 Bit 5 Bit 6 Bit 7 Bit 8 – Total Bit = 8

Binary Conversion for 10 :

10 : 128 64 32 16 8 4 2 1

0 0 0 0 1 0 1 0

Bit 1 Bit 2 Bit 3 Bit 4 Bit 5 Bit 6 Bit 7 Bit 8 – Total Bit = 8

8.8.8.8 – total of 32 Bit.

Dotted Decimal Notation : In dot form 4 Sections are called as OCTETS – Vendor Neutral Term for Bytes.

Let a IP Be : 200.10.20.30

Inside a Network : 200.10.20 – will remain same and 30 will be unique for each.

Type of IP Address –

Assignment Method
Classes : 1) Classful
2) Classless
Public / Private
Version

Assignment Methods :

Assignment Method is method that defines how to assign an IP address to a Device.

IP Address can be assigned in two ways

1) Static IP Address

Static IP Address is the IP Address in which configuration is done Manually and is used in small networks.

2) Dynamic IP Address

Dynamic IP Address is the IP Address in which the configuration is done by the Computer Interface or by the Host Interface – DHCP (Dynamic Host Configuration Protocol)

— Configuration is Automatic–

Classes :

classes define that in an IP, How much part will be for Network and How much is for Host.

There are 2 types of classes in IP Addressing :

Classful
Classless

CLASSFUL : IP Address are divided into 5 Classes;

Class A : 0 – 126 N.H.H.H Assigned for Large Organization

127 N.H.H.H Assigned for the Loopback

Class B : 128 – 191 N.N.H.H Assigned for Medium Companies

Class C : 192 – 223 N.N.N.H Assigned for Small Organizations

Class D : 224 – 239 Assigned for Multicasting

Class E : 240 – 255 Assigned for Experimental Purpose

CLASSLESS : Classless addressing is an IP address where a subnet mask does not define its class. Subnet mask can be anywhere between bit 0 and bit 31.

CLASS A IP ADDRESS :

Range of Class A IP Address : 0.0.0.0 – 127.255.255.255

Network ID : 8 Bit

Host ID : 24 Bit (8+8+8)

IP Address begins with 0,First Bit will always be Zero
7 Remaining Bits in Network part : Only 128 Possible class A Network
24 Bits in Local Part : Over 16 million hosts per Class A Network
All class A network parts are assigned or reserved.

Network ID(N)

Host ID(H)

0 7 8 31

0NNNNNNN . HHHHHHHH . HHHHHHHH . HHHHHHHH

In Binary :

Class A starts from : 00000000.00000000.00000000.00000000

Class A ends at : 01111111.11111111.11111111.11111111

In Decimal :

Class A IP Address is from 0.0.0.0 to 127.255.255.255

Number of Networks : 2^7 = 128

Number of Hosts : 2^24

SOME EXCEPTIONS IN CLASS A : Cannot be assigned to host

0.0.0.0 : For Self check – Represent Default Network or M

0.255.255.255 : For Self check – Represent Default Network or My IP

127.0.0.0 : Loop Back Address Range : solve NIC Problem

127.255.255.255 : Loop Back Address Range : solve NIC Problem

CLASS B IP ADDRESS :

Range of Class B IP Address : 128.0.0.0 – 191.255.255.255

Network ID : 16 Bit(8+8)

Host ID : 16 Bit (8+8)

First two Bit will always be One and Zero
14 Bits in Network part – Over 16,000 possible Class B Network
16 Bits in Local Part – Over 65,000 possible Hosts

Network ID(N)

Host ID(H)

0 15 | 16 31

10NNNNNN . NNNNNNNN . HHHHHHHH . HHHHHHHH

In Binary :

Class B starts fr0m : 10000000.00000000.00000000.00000000

Class B ends at : 10111111.11111111.11111111.11111111

In Decimal :

Class B IP Address is from 128.0.0.0 to 191.255.255.255

Number of Networks : 2^14

Number of Hosts : 2^16

SOME EXCEPTIONS IN CLASS B : Cannot be assigned to host

169.254.X.X : Reserved for APIPA (Automatic Private IP Address) – Host take IP Automatically ifit doesn’t get any DHCP Server in the Network.

CLASS C IP ADDRESS :

Range of Class B IP Address : 192.0.0.0 – 223.255.255.255

Network ID : 24 Bit(8+8+8)

Host ID : 8 Bit (8)

**Most Popular and Commonly Used**

First three Bit will always be One,One and Zero
21 Bits in Network part – Over 2 Million possible Class C Network
8 Bits in Local Part – Only 256 possible Hosts per class C Network

Network ID(N)

Host ID(H)

0 23 | 24 31

110NNNNN . NNNNNNNN . NNNNNNNN . HHHHHHHH

In Binary :

Class C starts from : 1100000.00000000.00000000.00000000

Class C ends at : 11011111.11111111.11111111.11111111

In Decimal :

Class C IP Address is from 192.0.0.0 to 223.255.255.255

Number of Networks : 2^21

Number of Hosts : 2^8

CLASS D IP ADDRESS :

Range : 224.0.0.0 – 239.255.255.255
IP Address begins with 1110

Used for Multicasting, Not defining networks.

Sending messages to group of hosts
just to one (Unicasting)
ALL HOSTS (Broadcasting)
Say to send a videoconference stream to a group of receivers

In Binary :

Class D starts from : 11100000.00000000.00000000.00000000

Class D end at : 11101111.11111111.11111111.11111111

In Decimal :

Class D IP Address is from 224.0.0.0 to 239.255.255.255

224.0.0.5 – OSPF
All OSPF Routers address is used to send HELLO PACKETS

224.0.0.6 – OSPF
All the routers address is used to send OSPF routing information to designated routers on a network segment.

224.0.0.9 – The Routing Information Protocol (RIP) version 2 group address is used to send routing information to all RIP2-aware routers on a network segment.

224.0.0.10 – EIGRP
used to send routing information to all EIGRP routers on a network segment.

224.0.0.18 – Virtual Router Redundancy Protocol.

Private/Public:

PUBLIC :

A public also called as External IP address is the one that your ISP (Internet Service Provider) provides to identify your home network to the outside world. It is an IP address that is unique throughout the entire Internet.

When you’re setting up your router, if your ISP issued you a static IP address, you enter it into your router’s settings. For a dynamic IP address, you specify DHCP in your router’s network settings. DHCP is Dynamic Host Control Protocol. It tells your router to accept whatever public IP address your ISP issues.

Those who wanted not to connect through internet but they wanted to run their network on TCP/IP Protocol

Here came the concept of PRIVATE IP

PRIVATE :

Just as your network’s public IP address is issued by your ISP, your router issues private (or internal) IP addresses to each network device inside your network. This provides unique identification for devices that are within your home network, such as your computer, your Slingbox, and so on.

THEY ARE NOT ROUTABLE

CLASS A PRIVATE ADDRESS 10.0.0.0 – 10.255.255.255

CLASS B PRIVATE ADDRESS 172.16.0.0 – 172.31.255.255

CLASS C PRIVATE ADDRESS 192.168.0.0 – 192.168.255.255

Internet Protocol Address :

Reserved IP Address :

Addresses beginning with 127 are reserved for loopback and internal testing – Used for Self Testing that TCP/IP is properly working or not.
XXX.0.0.0 reserved for Network Address
XXX.255.255.255 reserved for Broadcast
0.0.0.0 – First Address – Represent Local Network / Used for Default Routing
255.255.255.255 – Broadcast

Example : Let a Class A IP Address be – 101.101.101.101

Network Address – 101.0.0.0

BroadCast Address – 101.255.255.255

: Let a Class B IP Address be – 150.150.150.150

Network Address – 150.150.0.0

BroadCast Address – 150.150.255.255

I hope that gives you a good knowledge of IP Addresses and their classes.

Now, We can move on to what sub-netting is, in my next blog.

Please Follow this link to get on to sub-netting –
Classless Inter Domain Routing Made Easy (Cont..)

Snoopy + ELK : Exhibit sudo commands in Kibana Dashboard

Logging User Commands: Snoopy Logger

About Snoopy Logger

Snoopy logs all the commands that are ran by any user to a log file. This is helpful for auditing and keep an eye on user activities.

Automated Installation

For Automated Installation/Configuration of Snoopy we have created a Puppet module and Ansible Role.

Puppet Module: https://forge.puppetlabs.com/opstree/snoopy

Ansible Role: https://galaxy.ansible.com/OpsTree/Snoopy/

Manual Installation

To install the latest STABLE version of Snoopy, use these commands:

rm -f snoopy-install.sh
wget -O snoopy-install.sh https://github.com/a2o/snoopy/raw/install/doc/install/bin/snoopy-install.sh
chmod 755 snoopy-install.sh
./snoopy-install.sh stable

Output

This is what typical Snoopy output looks like:

2015-02-11T19:05:10+00:00 labrat-1 snoopy[896]: [uid:0 sid:11679 tty:/dev/pts/2 cwd:/root filename:/usr/bin/cat]: cat /etc/fstab.BAK
2015-02-11T19:05:15+00:00 labrat-1 snoopy[896]: [uid:0 sid:11679 tty:/dev/pts/2 cwd:/root filename:/usr/bin/rm]: rm -f /etc/fstab.BAK

These are default output locations on various Linux distributions:

CentOS: /var/log/secure
Debian: /var/log/auth.log
Ubuntu: /var/log/auth.log
others: /var/log/messages (potentially, not necessarily)

For actual output destination check your syslog configuration.

Snoopy provides a configuration file “/etc/snoopy.ini” where you can configure snoopy to generate logs. By default snoopy logs only uid, but doesn’t logs username in logs, so we have to change configuration to get username in logs.You may also specify the log path where you want to generate the snoopy logs.

For getting username in logs edit “/etc/snoopy.ini” and under [snoopy] section add the following line:

message_format = “[username:%{username} uid:%{uid} sid:%{sid} tty:%{tty} cwd:%{cwd} filename:%{filename}]: %{cmdline}”

The output of logs is shown below:

Feb 25 07:47:27 vagrant-ubuntu-trusty-64 snoopy[3163]: [username:root uid:0 sid:1828 tty:/dev/pts/0 cwd:/root filename:/usr/bin/vim]: vim /etc/snoopy.ini

Enable/Disable Snoopy

To enable snoopy, issue the following command:

snoopy-enable

To disable snoopy, issue the following command:

snoopy-disable

Using ELK to parse logs

Now that we have logs with suitable information we will write a grok pattern in logstash to parse these logs and generate required fields.

A sample grok pattern will be like this:

filter {

if [type] == “snoopy” {

grok {

match => { “message” => “%{SYSLOGTIMESTAMP:date} %{HOSTNAME:hostname} %{WORD:logger}\[%{INT}\]\: \[%{WORD}\:%{USERNAME:username} %{DATA} %{DATA} %{DATA} %{WORD}\:%{DATA:cwd} %{DATA}\]\: %{GREEDYDATA:exe_command}” }

}

if “_grokparsefailure” in [tags] {

drop { }

}

Here we are generating these fields:

date: Timestamp at which log is generated

hostname: Name of host

logger: Name of logger which is generating logs in our case “snoopy”.

username: Name of user issuing the command

cwd: Absolute path of directory from where the command is executed

exe_command: Command that is executed by user with complete options

Place the above grok pattern in filter section of logstash configuration file which is at “/etc/logstash/conf.d/logstash.conf”. Also include logs from “/var/log/auth.log” to be shipped to logstash server from logstash agent at the client.

Creating Dashboard in Kibana

After that you can see these logs in kibana in “Discover” tab as shown in screenshot:

In the left sidebar you can see all the fields via which you can filter including the fields we set in our grok pattern.Now in the search bar you can search according to specific field and its value. For example to search logs for vagrant user and all sudo commands executed by it, you will write the following query in search bar:

username:vagrant AND exe_command:sudo*

Then from the left sidebar add the fields you want to see, for example add “username”, “exe_command” and “cwd”, which will result to a table as shown below:

Now save this search from the icon that is just adjacent to left bar with a suitable name. Then go to “Dashboard” menu and click on “plus” icon to add a dashboard. A screen will appear as shown:

Click on “Searches” tab and find your saved search and click over it. A resulting screen will appear which will be added to your dashboard as shown below:

Here you can view tabular data for the sudo commands executed by vagrant user. Similarly you can add more searches by clicking on “plus icon” and add it to the same dashboard.Now save this dashboard by clicking on the “save” icon adjacent to search bar with a suitable name.After that you can easily load this dashboard by clicking on “load” icon adjacent to search bar.

Snoopy : Get, Set and log

Recently we got a requirement to log each and every command executed by the users. Upon googling we found a tool that is exclusively contrived to accomplish this ambition.

This tool was “Snoopy”. This is a open source project whose microscopic documentation is available at their github page:

https://github.com/a2o/snoopy

You can find all installation and configuration doc at their github README.

Why snoopy?

We choose snoopy because it is a very lightweight tool that is just built with the intention of logging each and every command executed by user. It consists of only a tiny library which does all the logging.

Use cases:

1). Log each executed command by any user with its arguments

2). Configure the pattern of logs according to requirement using a simple configuration file

3). The logs generated by snoopy can be sent to a central logging server and then can be parsed to get useful information. For example: To get all sudo commands executed by a user.

About Snoopy:

Snoopy logger is a majestic utility which makes the admin work more effortless by providing a log of commands with its arguments executed via shell by any user. It comes with a configuration file “/etc/snoopy.ini” where you can configure how the logs are generated and the information in the logs you want to have.

Automating Snoopy installation/configuration:

We will make the procedure easy for installation and configuration of snoopy by creating puppet module and ansible role. We’ll soon be launching a puppet module and an ansible role for it.

Also we’ll showcase a useful example of snoopy logger with centralized logging using ELK.