Skip to main content

Snoopy + ELK : Exhibit sudo commands in Kibana Dashboard


Logging User Commands: Snoopy Logger

About Snoopy Logger

Snoopy logs all the commands that are ran by any user to a log file. This is helpful for auditing and keep an eye on user activities.

Automated Installation

For Automated Installation/Configuration of Snoopy we have created a Puppet module and Ansible Role.

Manual Installation

To install the latest STABLE version of Snoopy, use these commands:

rm -f snoopy-install.sh
wget -O snoopy-install.sh https://github.com/a2o/snoopy/raw/install/doc/install/bin/snoopy-install.sh
chmod 755 snoopy-install.sh
./snoopy-install.sh stable

Output

This is what typical Snoopy output looks like:

2015-02-11T19:05:10+00:00 labrat-1 snoopy[896]: [uid:0 sid:11679 tty:/dev/pts/2 cwd:/root filename:/usr/bin/cat]: cat /etc/fstab.BAK
2015-02-11T19:05:15+00:00 labrat-1 snoopy[896]: [uid:0 sid:11679 tty:/dev/pts/2 cwd:/root filename:/usr/bin/rm]: rm -f /etc/fstab.BAK

These are default output locations on various Linux distributions:
  • CentOS: /var/log/secure
  • Debian: /var/log/auth.log
  • Ubuntu: /var/log/auth.log
  • others: /var/log/messages (potentially, not necessarily)
For actual output destination check your syslog configuration.

Snoopy provides a configuration file “/etc/snoopy.ini” where you can configure snoopy to generate logs. By default snoopy logs only uid, but doesn’t logs username in logs, so we have to change configuration to get username in logs.You may also specify the log path where you want to generate the snoopy logs.

For getting username in logs edit “/etc/snoopy.ini” and under [snoopy] section add the following line:
message_format = "[username:%{username} uid:%{uid} sid:%{sid} tty:%{tty} cwd:%{cwd} filename:%{filename}]: %{cmdline}"

The output of logs is  shown below:
Feb 25 07:47:27 vagrant-ubuntu-trusty-64 snoopy[3163]: [username:root uid:0 sid:1828 tty:/dev/pts/0 cwd:/root filename:/usr/bin/vim]: vim /etc/snoopy.ini

Enable/Disable Snoopy

To enable snoopy, issue the following command:
snoopy-enable
To disable snoopy, issue the following command:
snoopy-disable

Using ELK to parse logs

Now that we have logs with suitable information we will write a grok pattern in logstash to parse these logs and generate required fields.
A sample grok pattern will be like this:

filter {
 if [type] == "snoopy" {
   grok {
     match => { "message" => "%{SYSLOGTIMESTAMP:date} %{HOSTNAME:hostname} %{WORD:logger}\[%{INT}\]\: \[%{WORD}\:%{USERNAME:username} %{DATA} %{DATA} %{DATA} %{WORD}\:%{DATA:cwd} %{DATA}\]\: %{GREEDYDATA:exe_command}" }
   }
 if "_grokparsefailure" in [tags] {
   drop { }
 }
 }
}

Here we are generating these fields:
date: Timestamp at which log is generated
hostname: Name of host
logger: Name of logger which is generating logs in our case “snoopy”.
username: Name of user issuing the command
cwd: Absolute path of directory from where the command is executed
exe_command: Command that is executed by user with complete options

Place the above grok pattern in filter section of logstash configuration file which is at “/etc/logstash/conf.d/logstash.conf”. Also include logs from “/var/log/auth.log” to be shipped to logstash server from logstash agent at the client.

Creating Dashboard in Kibana

After that you can see these logs in kibana in “Discover” tab as shown in screenshot:
elkdiscover.png

In the left sidebar you can see all the fields via which you can filter including the fields we set in our grok pattern.Now in the search bar you can search according to specific field and its value. For example to search logs for vagrant user and all sudo commands executed by it, you will write the following query in search bar:
username:vagrant AND exe_command:sudo*
Then from the left sidebar add the fields you want to see, for example add “username”, “exe_command” and “cwd”, which will result to a table as shown below:

elktableselectedfields.png

Now save this search from the icon that is just adjacent to left bar with a suitable name. Then go to “Dashboard” menu and click on “plus” icon to add a dashboard. A screen will appear as shown:

adddashboard.png

Click on “Searches” tab and find your saved search and click over it. A resulting screen will appear which will be added to your dashboard as shown below:

dashboardadded.png

Here you can view tabular data for the sudo commands executed by vagrant user. Similarly you can add more searches by clicking on “plus icon” and add it to the same dashboard.Now save this dashboard by clicking on the “save” icon adjacent to search bar with a suitable name.After that you can easily load this dashboard by clicking on “load” icon adjacent to search bar.

Comments

Popular posts from this blog

EC2 Ssh Connection Refused

When ssh: connect to host ip_address port 22 Connection refused



Unable to access server???
Exactly when you see the error - “ssh: connect to host ip_address port 22: Connection refused” while connecting your AWS EC2 Instance. In order to find solution of the problem, you will go to AWS forum and other channels where you need to answers several questions first. But it's very difficult to find the actual problem. In order to get clues what the problem is, we should provide as many details as possible about what we have tried and the results we are getting. Because there are hundreds of reason why a server or service might not be accessible, also connectivity is one of the toughest issue to diagnose, especially when you are hosting something critical on your box. I've seen several topics on this problem, but none offers a solution to it.  I was not aware for what should I look at first. So I walk through from the very basics and investigated the following thing Use of verbose while ss…

jgit-flow maven plugin to Release Java Application

Introduction As a DevOps I need a smooth way to release the java application, so I compared two maven plugin that are used to release the java application and in the end I found that Jgit-flow plugin is far better than maven-release plugin on the basis of following points: Maven-release plugin creates .backup and release.properties files to your working directory which can be committed mistakenly, when they should not be. jgit-flow maven plugin doesn't create these files or any other file in your working directory.Maven-release plugin create two tags.Maven-release plugin does a build in the prepare goal and a build in the perform goal causing tests to run 2 times but jgit-flow maven plugin builds project once so tests run only once.If something goes wrong during the maven plugin execution, It become very tough to roll it back, on the other hand jgit-flow maven plugin makes all changes into the branch and if you want to roll back just delete that branch.jgit-flow maven plugin doesn…

VPC per envrionvment versus Single VPC for all environments

This blog talks about the two possible ways of hosting your infrastructure in Cloud, though it will be more close to hosting on AWS as it is a real life example but this problem can be applied to any cloud infrastructure set-up. I'm just sharing my thoughts and pros & cons of both approaches but I would love to hear from the people reading this blog about their take as well what do they think.


Before jumping right away into the real talk I would like to give a bit of background on how I come up with this blog, I was working with a client in managing his cloud infrastructure where we had 4 environments dev, QA, Pre Production and Production and each environment had close to 20 instances, apart from applications instances there were some admin instances as well such as Icinga for monitoring, logstash for consolidating logs, Graphite Server to view the logs, VPN server to manage access of people.




At this point we got into a discussion that whether the current infrastructure set-u…