Shell initialization files

Introduction

A shell initialization file is a shell script that runs automatically each time the shell executes. The initialization file sets up the “work environment” and “customizes” the shell environment for the user. The main agenda of Shell initialization files are to persist common shell configuration, such as:

  • $PATH and other environment variables
  • shell prompt
  • shell tab-completion
  • aliases, functions
  • key bindings

Shell modes

Before understanding shell initialization files, first understand shell initialization files are dependent on the “combination of shell modes” in which a particular shell process runs. The shell can be run in three possible modes:

  • Interactive login 
  • Interactive non-login 
  • Non-interactive 

Here are some operations which result in the execution of different combination shell modes

  • Login to a remote system via SSH : Login, Interactive
  • User successfully login into the system, using /bin/login, after reading credentials stored in the /etc/passwd file: Login, Interactive
  • Execute a script remotely and request a terminal, e.g. ssh user@host -t ‘echo $PWD’ : Non-Login, Interactive
  • Start a new shell process, e.g. bash : Non‑Login, Interactive
  • Execute a script remotely, e.g. ssh user@host ‘echo $PWD : Non‑Login, Non‑Interactive
  • Run a script, bash myscript.sh: Non‑Login, Non‑Interactive
  • Run an executable with #!/usr/bin/env bash shebang : Non‑Login, Non‑Interactive
  • Open a new graphical terminal window/tab: Non‑Login, Interactive

Now, back to our shell initialization files execution based on shell modes.

There are two types of shell initialization files 

  • System-wide startup files
  • User-specific startup files

System-wide Startup Files

These are the initialization files that contain configurations that applied to the whole system irrespective of a specific user, which means all users can share the same configuration which applied in system-wide startup files. System-wide startup files are:

  • The /etc/profile file – It stores system-wide environment configurations and startup programs for login setup. All configurations that you want to apply to all system users’ environments should be added in this file.
  • The /etc/bashrc or /etc/bash.bashrc file – It contains system-wide functions and aliases including other configurations that apply to all system users.

User-specific startup files

These are the initialization files which contain configuration which applied to the specific user, means all users can have their own configuration which applied in user-specific startup files. User-specific startup files are located in home directory of the user and files are .profile, .bash_profile, .bashrc and .bash_login.

  • ~/.bash_profile file – Stores user-specific environment and startup programs configurations. 
  • ~/.bashrc file – Stores user-specific aliases and functions.
  • ~/.bash_login file – Contains specific configurations that are normally only executed when you log in to the system. When the ~/.bash_profile is absent, this file will be read by bash.
  • ~/.profile file – It is read in the absence of ~/.bash_profile and     ~/.bash_login; it can store the same configurations, which are can also be accessible by other shells on the system. Because we have mainly talked about bash here, take note that other shells might not understand the bash syntax.
  • ~/.bash_history file – Bash maintains a history of commands that have been entered by a user on the system. This list of commands is kept in a user’s home directory in the ~/.bash_history file.
  • ~/.bash_logout file – it’s not used for shell startup, but stores user specific instructions for the logout procedure. It is read and executed when a user exits from an interactive login shell.

Order of activation of system-wide startup files and user-specific startup files based on shell mode:

1. Non-interactive mode:

source file in $BASH_ENV

2. Interactive login mode:

  • /etc/profile
  • ~/.bash_profile, ~/.bash_login, ~/.profile (only first one that exist)

3. Interactive non-login mode:

  • /etc/bash.bashrc
  • ~/.bashrc

Now we have a good idea of what are all the shell modes, why shell initialization files are important and how the execution of shell initialization files depends on the different combination shell modes.

With this knowledge, we can achieve modification to shell and access to users based on the shell mode.

Reference:

Image – https://blog.flowblok.id.au/2013-02/shell-startup-scripts.html

Blog – https://www.tecmint.com/understanding-shell-initialization-files-and-user-profiles-linux/

What, Why and How of CTF Challenges?

What is CTF?

Capture The Flag challenge, better known as CTF, is an Information Security competition that requires contestants to exploit a machine or piece of code to extract specific pieces of text that may be hidden in a web page or a server known as the flag. It can comprise of many challenges across various genres such as Reverse Engineering, Networks and Protocols, Programming, Crypto, Web Security, Exploits, etc. All these puzzles have just one goal, capture the flag! 

Why should you CTF often?

Why You Should Start Doing CTFs? Capture The Flag (CTF) is a competition in the Information Security field. The main idea is to simulate different kinds of attack concepts with various challenges, which eventually opens your mind to look at things from a different perspective no matter which side of infrastructure you are on. CTFs generally expose you to things You Wouldn’t Learn Otherwise. 

If you’re a developer it would teach you how to write secure code. If you’re an Ops/DevOps professional, it would teach you to setup your infrastructure and practices around it in the most secure manner. 

So, Why shouldn’t you do it? Plus it’s kinda fun too! 

How should you begin ?

Coming to the most important section of this blog with hope that you are slightly convinced to pick up CTF for once at least. Great! 

So let’s focus on the how and where part of CTFs now. 

Firstly, here you can find a list of all the CTFs going around the world, be it offline or online. 

Here you’ll see that there are mostly two types of CTF organized. Maximum times, it’s the Jeopardy style CTF where you are provided a list of challenges and award points to individuals or teams that complete the challenges; groups with the most points wins.

And sometimes if you’re lucky you’ll find Attack-Defense style CTF where CTFs focus on either attacking an opponent’s servers at the same time defending one’s own. You get points for attacking other teams and points are deducted every time an opponent attacks your system.

A good place to start for various genres are listed below : 

→ Linux : https://overthewire.org/wargames/leviathan/ , https://overthewire.org/wargames/bandit/

→ Web : https://owasp.org/ , https://sourceforge.net/projects/owaspbwa/ https://overthewire.org/wargames/natas/

→ Reverse Engineering : https://www.root-me.org/ ,
https://crackmes.one/ http://reversing.kr/challenge.php

→ Binary Exploitation: https://ctf101.org/binary-exploitation/overview/

→ Pwn challenges : http://pwnable.kr/ , https://pwn0.com/

→ Cyber Forensics: https://ctf101.org/cryptography/overview/

Rest, Always have faith on google to show you the light when it gets dark! 

Feel free to join this group on Linkedin to learn and share more on security and DevOps

References : https://blog.usejournal.com/why-you-should-start-doing-ctfs-1c5e88eacabc

Recap Amrita InCTF 2019 | Part 2

+

Amrita InCTF 10th Edition is an offline CTF(Capture the Flag) event hosted by Amrita University. In our previous blog, we discussed about talks from the first day. In this we’ll share some lights on the talks from second day.

Talk 1: Exploring attack surfaces in embedded devices by Vivek 

The IoT’s has become popular in everyday household item like a fridge, washing machine, camera, and television. You can access them remotely and some devices can communicate with each other. These connections become entry points to the attacker. 

Now a days commodity devices are getting intelligent and SOC’s are pretty much cheaper($5 raspberry pi Zero),as such they are all over us from watch to contact glasses; these are all now getting connected to a massive IoT network all connected and potentially vulnerable.

Below are some insights on key topics discussed in the talk around IOT security.

BLE Security Testing:- Bluetooth Low Energy Low cost and ease of implementation lead BLE to be widely used among IoT devices and applications like Wearable sensors, light-bulbs and medical devices.

BLE has three main vulnerabilities.

  • Eavesdropping:- Eavesdropping refers to a third-party device listening in on the data that’s being exchanged between two paired devices.
  • Man in the Middle Attacks (MITM): Man in the middle attacks involve a third party device impersonating a legitimate device, tricking two legitimate devices into believing that they’re connected to each other, when in reality, the legitimate devices are connected to the impersonator.
  • Denial of Service & Fuzzing Attack: DoS attacks expose a system to the possibility of frequent crashes leading to a complete exhaustion of its battery.Fuzzing attacks too lead to systems crashing as an attacker may send malformed or non-standard data to a device’s.

ZigBee Security Testing- : Zigbee is a wireless communication Protocol. It’s used to connect the sensor, door locks, electric meters and traffic management systems. This protocol is open at a network level. So when the devices start connecting they send out beacon requests.

 NFC RFID cloning :- 

There is no security, they’ve been hacked, there’s no protection of data, no privacy, everything is in the clear and it’s not resistant to sniffing or common attacks. 

To access sensitive information, you have to provide that sector of memory with the right key—otherwise, it will show up blank. Even though these cards are a lot more secure, once you know the encryption algorithm you can decrypt them and access the sensitive information. With that, people can also clone these cards relatively easily.

Hello barbie hardware hacking:-  The doll uses Wi-Fi transmit audio from children talking like siri or google assistant. The toy uses a digital ID that attackers can abuse and potentially let them spy on the chatter between a doll and a server. phones with the app will automatically connect to any Wi-Fi network that includes “Barbie” in its name.

Talk 2: APT attack by Shaunak :

Shaunak is the CEO of Zacco cyber security company. He talks about his experience in APT attacks. He give a brief intro about what Advanced persistent threat attacks. He also shared his experience of finding out an APT attack within an organization which had no clue about it.

What is APT Attack:- 

APT attack are perform in large scale . APT attacks are a cyber crime directed at business and political targets. Organized crime groups may sponsor advanced persistent threats to gain information they can use to carry out criminal acts for financial gain.

How an APT attack works:-

  • Gain access: APT groups Target by targeting system. Like social engineering techniques and application vulnerability. 
  • Establish a foothold: After gaining access target, APT group do future reconnaissance, create networks of backdoors and tunnels that they can use to move around unnoticed. APTs may use advanced malware techniques such as code rewriting to cover their tracks.
  • Gain even greater access: Once inside a network APT actors may use such methods as password cracking to gain administrative rights. So they can get high level access.
  • Move laterally: After getting admin access, they can then move around the enterprise network.  They can attempt to attack other servers. 
  • Stage the attack: At this point, the hackers centralize, encrypt and compress the data so they can exfiltrate it.
  • Take the data: The attacker transfers data in his own system.
  • Remain until they’re detected: The APT group can repeat this process for a long time until they detected. 

 Talk 3: Intel L1 Terminal Fault Vulnerability
by Reno Robert

Reno Robert talks about Inlet L1 Terminal Fault most Intel processors are affected with this vulnerability. It can allow attackers to access sensitive information stored in the Level 1 CPU cache. 

This may include data from the operating system, kernel, hypervisor or the neighboring virtual machine.

It may allow a malicious code execution on one thread to access data from the L1 cache of another thread within the same core.

  • L1TF system information:- An attacker can use this vulnerability to read any physical memory location that is cached in the L1 data cache of the processor.
  • Page-table entries:- The memory addresses used by both user space and the kernel do not point directly into physical memory. Instead, the hierarchical page-table structure is used to translate between virtual and physical addresses. 
  • Flush L1 data cache on security domain transition:- The L1D is shared by all LPs on the same physical core. This means disclosure can be prevented by flushing the L1 data cache when transitioning between security domains. Intel has provided new capabilities through a microcode update that supports an architectural interface for flushing the L1 data cache.

This was all from day 2 talk, Come back on next Tuesday for talks from Day 3. And as the final segment of this series we’ll be updating about attack/defense and jeopardy CTF experience.

We’ll be more than happy to hear from you in comments section regarding any feedback or criticism.

Stay Tuned, Happy Blogging!

Reference: https://blog.attify.com/the-practical-guide-to-hacking-bluetooth-low-energy/

https://msrc-blog.microsoft.com/2018/08/14/analysis-and-mitigation-of-l1-terminal-fault-l1tf/

https://research.kudelskisecurity.com/2017/11/21/zigbee-security-basics-part-3/

Docker Inside Out – A Journey to the Running Container

Introduction: 

Necessity is the mother of invention, same happens here in case of docker. With the pressure of splitting monolithic applications for the purpose of ease, we arrived at docker and it made our life much simpler. We all access docker with docker-cli command but I wonder what it does behind the scene, to run a container. Let’s get deeper into it in this very blog.

There’s a saying that “Behind every successful man, there is a woman”. I would love to give my perspective on this. One of the things that I have actively observed from the life of successful people I know is that there is a lot of truth in this statement but it varies with different situations and in most of the cases these women are not directly helping men in their prime work but taking care of other important work around so that they can concentrate on their prime work. Keeping this in my mind, I am expecting that there are other components as well which are behind docker-cli command that leads to the successful creation of containers. Whenever I talk about docker containers with developers who are new to docker in my organization the only thing I hear from them is “docker-cli command is used to invoke docker daemon to run container”

But, Docker daemon is not the process that gets executed when a container is meant to be run – it delegates the action to containerd which then controls a list of runtimes (runc by default) which is then responsible for creating a new process (calling the defined runtime as specified in the configuration parameters) with some isolation and only then executing the entrypoint of that container.

Components involved

  • Docker-cli
  • Dockerd
  • Containerd
  • RunC
  • Containerd-shim

Docker-cli: Used to make Docker API calls.

Dockerd:  dockerd listens for Docker API requests, dockerd can listen for Docker Engine API requests via three different types of Socket: unix, tcp, and fd and manages host’s container life-cycles with the help of containerd. Hence, actual container life-cycle management is outsourced to containerd. 

Containerd: Actually manages container life-cycle through the below mentioned tasks:

  • Image push and pull
  • Management of storage
  • Of course executing containers by calling runc with the right parameters to run containers.

Let’s go through some subsystems of containerd :

Runc: Containerd uses RunC to run containers according to the OCI specification.

Containerd shim: With docker 1.11 version, this component has been added. This is the parent process of every container started and it also allows daemon-less containers. First it allows the runtimes, i.e. runc, to exit after which it starts the container.  This way we don’t have to have the long running processes for containers.  When you start nginx you should only see the nginx process and the shim.  

Daemon-less: When I say daemon-less containers in the above paragraph, it means there is an advantage of this. When containerd shim was not there, upgrading docker daemon without restarting all your containers was a big pain. Hence, containerd shim got introduced to solve this problem.

The communication between Dockerd and ContainerD

We can see how docker delegates all the work of setting up the container to containerd. Regarding interactions between docker, containerd, and runc, we can understand that without even looking at the source code – plain strace and pstree can do the job.

Command:

when no containers running:

ps fxa | grep docker -A 3 

Result:

Working of all components together

Well, To see how all these components work together? We need to initialize a container i.e. Nginx in our case. We will be firing the same command after running an Nginx container.

This shows us that we have two daemons running – the docker daemon and the docker-containerd daemon.

Given that dockerd interacts heavily with containerd all the time and the later is never exposed to the internet, it makes sense to bet that its interface is unix-socket based.

High-Level overview of initializing container

Initializing container to see involvement of all components

Command:

docker run --name docker-nginx -p 80:80 -d nginx
docker ps
pstree  -pg | grep -e docker -e containerd 
ps fxa | grep -i “docker” -A 3 | grep -v “java”

Summary

By now, it might be clear that dockerd is not only the single component involved while running a container. We got to know what all components are backing a running container beside dockerd and how they work together to manage the lifecycle of a container.

I hope we have a good understanding of the docker components involved. Now it’s time to see things practically on your own with commands discussed in this blog without mugging up theoretical concepts.

That’s all till next time, thanks for reading, I’d really appreciate your feedback, please leave your comment below if you guys have any feedback or any queries.

Happy Containerization !!

References :

Recap Amrita InCTF 2019 | Part 1

Amrita InCTF 10th Edition, is an offline CTF(Capture the Flag) event hosted by Amrita university at their Amritapuri campus 10 KM away from Kayamkulam in Kerala, India. In this year’s edition two people from Opstree got invited to the final round after roughly two months of solving challenges online. The dates for the final rounds were 28th,29th and 30th December 2019. The first two days comprised of talks by various people from the industry and the third day was kept for the final competition. In the upcoming three blog series starting now, we’d like to share all the knowledge, experiences and learning from this three day event.

Talk from Cisco

The hall was full of a little more than 300 people, among which a lot were college students all the way ranging from sophomore year up till final as well as pre-final year. Also, to our surprise there were roughly 50+  school students sitting ready to compete for the final event as well. The initial talk by CISCO was refreshing and very insightful for everyone present in the room. The talk majorly focused on how technology is changing lives all around the world be it with machine learning to help doctors treat faster or be it use drones to put off fire or IoT enabled system to provide efficient irrigation at remote areas. The speakers also made a point on how learning in a broader segment of technologies and tools serves longer than in depth knowledge of limited technology.
One thing that really stuck with me was that never learn a technology just for the sake of it or for the hype around it. But learn with a thought on how it can solve a problem around us.

Talk Title: Cyberoam startup and experiences -Hemanth Patel

Hemal Patel talked about his couple of startups and how he has always learned through failures. The talk was full of experiences and it is always serene to listen to someone telling about how they failed over and over again which eventually led them to succeed at whatever they are doing today. He talked about CyberRoam which is a Sophos Company, secures organizations with its wide range of product offerings at the network gateway. The talk went on to give us an overview of how business is done along different governments all around the world and how Entrepreneurship is so much more than just tackling a problem at a business level. And the how Cyberroam ended up making the product that they have today. 

Talk on Security by Cisco – Radhika Singh and Prapanch Ramamoorthy 

This was a wide range talk about a lot of things affecting us. We’ll try to list down most of it here. 

The talk started out with exploring Free/Open WiFi. Though it has a huge benefit of wifi being free it comes with a lot of risks as well. To name a few : 

→ Sniffing

→ Snooping 

→ Spoofing

These just to mention a few ways you can be compromised over a free WiFi. 

You can read up more on it here :

The talk also presented us with facts over data, how only 1% of the total data is generated via laptops and computers, Rest all are generated by smart phones, smart TVs as other IoT devices. Hence comes a very important point of securing IoT devices. 

It was pointed out during the talk that majority of the companies worry about security over the end of the entire IoT chain i.e. over the cloud etc. But not many people are caring about the edge devices and how lack of security measures here can compromise them. 

There was this really interesting case study about of IoT devices brought down the internet for the entire US east coast and how this attack was just meant to get some more time to submit an assignment at it’s initial days. Read more on this story from 2016 here

Hackers prefer to exploit IOT devices over cloud infrastructure.

Memes apart, The talk also focused on privacy vs security and how  Google’s dns resolution encryption helps in securing DNS based internet traffic on the world wide web. 

National Critical Information Infrastructure Protection Centre(NCIIPC)

National Critical Information Infrastructure Protection Centre (NCIIPC) is an organisation of the Government of India created under Sec 70A of the Information Technology Act, 2000 (amended 2008), through a gazette notification on 16th Jan 2014, based in New Delhi, India. It is designated as the National Nodal Agency in respect of Critical Information Infrastructure Protection. 

Representatives from this organization was there to speak at the event and they talked in detail about defining what is a CII (Critical Information Infrastructure) is and how any company with such infrastructure needs to inform the government about it. 

A CII is basically any Information Infrastructure (by any financial/medical etc institute) which if compromised can affect the national security of the country. And attacking any such infrastructure is an act of terrorism as defined by the article 66F in the IT Act,2018. 

They talked about some of the threats they deal with at the national level. They particularly talked about how BGP routing protocol which works on trust was compromised lately to route all Indian traffic via Pakistan servers/routers.

Image result for darknet dark web internet

One more interesting talk was about the composition of Internet.

How we think that the internet we see would comprise of 90% of the total internet but in reality it’s just 4%, bummer right? .  Deep web is the one which comprises of 90% of the total internet and as a matter of fact that no one completely knows about the DarkNet and it’s volume. Hence even the numbers mentioned above are as good as a guess.

 This was a very insightful talk and put a lot of things in perspective.

Digital Forensics – Beyond the Good Ol’ Data Recovery by Ajith Ravindran 

Image result for data forensics

This talk by Ajith Ravindran mainly focused on Computer forensics, which is the application of investigation and analysis techniques to gather and preserve evidence from a particular computing device in a way that is suitable for presentation in a court of law.

The majority of tips and tricks shared were about getting data from Windows based machines even after it is deleted from the system and how such data can be retrieved in order to show as proof for crimes.

Some of the tricks talked about are mentioned below : 

The prefetch files in Windows  gives us the list of files and executables last accessed and the number of times executed. 

Userassist allows investigators to see what programs were recently executed on a system.

Shellbags list down files that are accessed via a user at least once.

Master file table enables us to get a list of all the files in the system, or even entered the system via network of USB drives.

$usrnjrnl gives us information regarding all user activities in past 1-2 days.

Hiberfil.sys is a file the system creates when the computer goes into hibernation mode. Hibernate mode uses the Hiberfil.sys file to store the current state (memory) of the PC on the hard drive and the file is used when Windows is turned back on.

This was all from day 1 talk, Come back on next Tuesday for talks from Day 2. And as the final segment of this series we’ll be updating about attack/defense and jeopardy CTF experience.

Stay Tuned, Happy Blogging!

Log Everything as JSON

Logging and monitoring are like Tony Stark and his Iron Man suit, the two will go together. Similarly, logging and monitoring work best together because they complement each other well.

For many years, logs have been an essential part of troubleshooting application and infrastructure performance. But over the period of time we have realized that logs are not only meant for troubleshooting purposes, they can also be used for business dashboards visualization and performance analysis.

So logging application data in a file is great, but we need more.

Why JSON logging is best framework?

For understanding the greatness of the JSON logging framework, let’s understand this conversation between Anuj(A system Engineer) and Kartik(A Business Analyst).


A few days later Kartik complains that Web Interface is broken. Anuj scratches his head and takes a look at the logs and realizes that Developer has added an extra field to the log lines broked his custom parser.

I am sure anyone can face a similar kind of situation.

In this case, if Developer has designed the application to write logs as JSON, it would be a piece of cake for Anuj to create a parser for that because then he has to search fields on the basis of the JSON key and it doesn’t matter how many new fields are getting added in the logline.

The biggest benefit of logging in JSON is that it has a structured format. This makes possible to analyze application logs just like Big Data. It’s not just readable, but a database that can be queried for each and every field. Also, every programming language can parse it.

Magic with JSON logging

Recently, we have created a sample Golang application to get Code Build, Code Test and Deployment phase experience with Golang Applications. So while writing this application we have incorporated the functionality to write logs in JSON.
The sample logs are something like this:-

And while integrating ELK for logs analysis, the only parsing line we have to add in logstash is:-

 
filter {
    json {
        source => "message"
    }
}

After this, we don’t require any further parsing and we can add as many fields in the log file.

As you can see I have all fields available in Kibana like:- employee name, employee city and for this, we do not have to add some complex parsing in logstash or in any other tool. Also, I can create a beautiful Business Dashboard with this data.

Application Repository Link:-
https://github.com/opstree/Opstree-Go-WebApp

Conclusion

It will not take too long to migrate from text logging to JSON logging as there are multiple programming language log drivers are available. I am sure JSON logging will provide more flexibility to your current logging system.
If your organization is using any Log Management platform like Splunk, ELK, etc. I think JSON logging could be a companion of it.

Some of the popular logging drivers which support JSON output are:-

Golang:- https://github.com/sirupsen/logrus
Python:- https://github.com/thangbn/json-logging-python
Java:- https://howtodoinjava.com/log4j2/log4j-2-json-configuration-example/
PHP:- https://github.com/nekonomokochan/php-json-logger

I hope now we have a good understanding of JSON logging. So now it’s time to choose your logging wisely.


That’s all I have, thanks for reading, I’d really appreciate any and all feedback, please leave your comment below if you guys have any feedback or any queries.

Cheers till next time!!

Image Reference:- https://www.google.com/url?sa=i&source=images&cd=&cad=rja&uact=8&ved=2ahUKEwi0jJzD8t_mAhVOzDgGHc6ODNQQjB16BAgBEAM&url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DbPTZ43nM688&psig=AOvVaw0N3k7-bSq4OD4oMxPxYPrN&ust=1577881894317756

How to test Ansible playbook/role using Molecules with Docker

Why Molecule?

Have you ever faced issue that your Ansible code gets executed successfully but something went wrong like, service is not started, the configuration is not getting changed, etc?

There is another issue which you might have faced, that your code is running successfully on Redhat 6 but not running successfully on Redhat 7, to make your code smart enough to run on every Linux flavour, in order to achieve this, molecule came into the picture. Let’s start some brainstorm in Molecule.

Molecule has capability to execute YML linter and custom test cases which you have written for your Ansible code. We will explain the linter and test cases below

Why code testing is required?

Sometimes during the playbook execution, although it executes playbook fine but it does not give us the desired result so in order to check this we should use code testing in Ansible.

In general, code testing helps developer to find bugs in code/application and make sure the same bugs don’t cause the application to break. it also helps us to deliver application/software as per standard of code. code testing helps us to increase code stability.

Introduction :

This whole idea is all about to use Molecule (A testing tool) you can test your Ansible code whether it’s functioning correctly on all Linux flavour including all its functionalities or not.

Molecule is a code linter program that analyses your source code for potential errors. It can detect errors such as syntax errors; structural problems like the use of undefined variables, etc.

The molecule has capabilities to create a VM/Container environment automatically and on top, it will execute your ansible code to verify all its functionalities.

Molecule Can also check syntax, idempotency, code quality, etc

Molecule only support Ansible 2.2 or latest version

NOTE: To run ansible role with the molecule in different OS flavour we can use the cloud, vagrant, containerization (Docker)
Here we will use Docker……………………

Let’s Start……………

How Molecule works:

“When we setup molecule a directory with name “molecule” creates inside ansible role directory then it reads it’s main configuration file “molecule.yml” inside molecule directory. Molecule then creates platform (containes/Instances/Servers) in your local machine once completed it executes Ansible playbook/role inside newly created platform after successful execution, it executes test cases. Finally Molecule destroy all newly created platform”

Installation of Molecule:

Installation of the molecule is quite simple.

$ sudo apt-get update
$ sudo apt-get install -y python-pip libssl-dev
$ pip install molecule [ Install Molecule ]
$ pip install --upgrade --user setuptools [ do not run in case of VM ]

That’s it…………

Now it’s time to setup Ansible role with the molecule. We have two option to integrate Ansible with molecule:

  1. With New Ansible role
  2. with existing Ansible role

1. Setup new ansible role with molecule:

$ molecule init role --role-name ansible-role-nginx --driver-name docker

When we run above command, a molecule directory will be created inside the ansible role directory

2. Setup the existing ansible role with molecule:

Goto inside ansible role and run below command.

$ molecule init scenario --driver-name docker

When we run above command, a molecule directory will be created inside the ansible role directory

NOTE: Molecule internally uses ansible-galaxy init command to create a role

Below is the main configuration file of the molecule:

  • molecule.yml – Contains the definition of OS platform, dependencies, container platform driver, testing tool, etc.
  • playbook.yml – playbook for executing the role in the vagrant/Docker
  • tests/test_default.py | we can write test cases here.

Content of molecule.yml

cat molecule/default/molecule.yml

---
molecule:
  ignore_paths:
    - venv

dependency:
  name: galaxy
driver:
  name: docker
lint:
  name: yamllint	
platforms:
  - name: centos7
    image: centos/systemd:latest
    privileged: True
  - name: ubuntu16
    image: ubuntu:16.04
provisioner:
  name: ansible
  lint:
    name: ansible-lint
#    enabled: False
verifier:
  name: testinfra
  lint:
    name: flake8
scenario:
  name: default  # optional
  create_sequence:
    - create
    - prepare
  check_sequence:
    - destroy
    - dependency
    - create

Explanation of above contents:

Dependency:

Testing roles may rely upon additional dependencies. Molecule handles managing these dependencies by invoking configurable dependency managers.

“Ansible Galaxy” is the default dependency manager.

Linter:

A linter is a problem which analyses our code for potential errors.

What code linters can do for you?

Code linter can do:

  1. Syntax errors;
  2. Check for undefined variables;
  3. Best practice or code style guideline.
  4. Extra lines.
  5. Extra spaces. etc

**We have linters for almost every programming languages like we have yamllint for YAML languages, etc.

yamllint: It checks for syntax validity, key repetition, lines length, trailing spaces, indentation, etc.

provisioner: Ansible is the default provisioner. No other provisioner will be supported.

Flake8:– is the default verifier linter. Usage python file

platforms:

What platform (Containers) will be created and Ansible code will be executed.

Driver:

Driver defines your platform where your Ansible code will be executed

Molecule supports below drivers:

  • Azure
  • Docker
  • EC2
  • GCE
  • Openstack
  • Vagrant

Scenario:

Scenario – scenario defines what will be performed when we run molecule

Below is the default scenario:

–> Test matrix

└── default
├── lint
├── destroy
├── dependency
├── syntax
├── create
├── prepare
├── converge
├── idempotence
├── side_effect
├── verify
└── destroy

However, we can change this scenario and sequence by changing molecule.yml file :

scenario:
  name: default  # optional
  create_sequence:      # molecule create 
    - create
    - prepare
  check_sequence:       # molecule check 
    - destroy
    - dependency
    - create
    - prepare
    - converge
    - check
    - destroy
  converge_sequence:    # molecule converge 
    - dependency
    - create
    - prepare
    - converge
  destroy_sequence:     # molecule destroy 
    - cleanup
    - destroy
  test_sequence:        # molecule test 
#    - lint
    - cleanup
    - dependency
    - syntax
    - create
    - prepare
    - converge

NOTE: If anyone scenario (action) fails, others will not be executed. this is the default molecule behaviour

Here I am defining all the scenarios:

lint: Checks all the YAML files with yamllint

destroy: If there is already a container running with the same name, destroy that container

Dependency: This action allows you to pull dependencies from ansible-galaxy if your role requires them

Syntax: Checks the role with ansible-lint

create: Creates the Docker image and use that image to start our test container.

prepare: This action executes the prepare playbook, which brings the host to a specific state before running converge. This is useful if your role requires a pre-configuration of the system before the role is executed.

Example: prepare.yml

---
- name: Prepare
  hosts: all
  gather_facts: false
  tasks:
    - name: Install net-tools curl
      apt: 
      	name: ['curl', 'net-tools']
      	state: installed 
      when: ansible_os_family == "Debian"

NOTE: when we run “molecule converge” below task will be performed :

====> Create –> create.yml will be called
====> Prepare –> prepare.yml will be called
====> Provisioning –> playbook.yml will be called

converge: Run the role inside the test container.

idempotence: molecule runs the playbook a second time to check for idempotence to make sure no unexpected changes are made in multiple runs:

side_effect: Intended to test HA failover scenarios or the like. See Ansible provisioner

verify: Run tests inside the container which we have written

destroy: Destroys the created container

NOTE: When we run molecule commands, a directory with name molecule created inside /tmp which is molecule managed, which contains ansible configuration, Dockerfile for all linux flavour and ansible inventory

cd /tmp/molecule

tree
.
└── osm_nginx
└── default
├── ansible.cfg
├── Dockerfile_centos_systemd_latest
├── Dockerfile_ubuntu_16_04
├── inventory
│ └── ansible_inventory.yml
└── state.yml

state.yml :- maintain scenario which has been performed .

Molecule managed

---
converged: true
created: true
driver: docker
prepared: true

Testing:

This is is most important part of Molecule where we will write some test cases.

Testinfra is the default test runner.

Below module should be installed:

  • $ pip install testinfra
  • $ molecule verify

Molecule calls below file for unit test using “testinfra” verifier

molecule/default/tests/test_default.py

verifier:

Verifier is used for running your test cases.

Below are the three verifiers which we can use in Molecule

  • testinfra – It usage python language for writing test cases.
  • goss – It usage yml language for writing test cases.
  • serverspac – usage ruby language for writing test cases.

Here I am using testinfra as verifier for writing test case.

Molecule commands:

  • # molecule check [ Run playbook.yml in check mode ]
  • # molecule create [ Create instance/ Platform]
  • # molecule destroy [ destroy instance / Platform]
  • # molecule verify [ perform unit test ]
  • # molecule test [ It performs below default scenario in sequence ]
  • # molecule prepare
  • #molecule converge

NOTE: To enable logs to run a command with –debug flag

$ molecule –debug test

Sample Test cases :

cat molecule/default/tests/test_default.py

import os

import testinfra.utils.ansible_runner

testinfra_hosts = testinfra.utils.ansible_runner.AnsibleRunner(
    os.environ['MOLECULE_INVENTORY_FILE']).get_hosts('all')

def test_user(host):
    user = host.user("www-data")
    assert user.exists

def test_nginx_is_installed(host):
    nginx = host.package("nginx")
    assert nginx.is_installed


def test_nginx_running_and_enabled(host):
  os = host.system_info.distribution
  if os == 'debian':
    nginx1 = host.service("nginx")
    assert nginx1.is_running
    assert nginx1.is_enabled

def test_nginx_is_listening(host):
    assert host.socket('tcp://127.0.0.1:80').is_listening

 That’s all ! we have covered all required topics which will help you to create your own environment of Molecule and test cases.

Thanks all !!! see you soon with new and effective blog 🙂

Links you may refer:

https://yamllint.readthedocs.io/en/stable/