Fasten Docker build

Gif for Fasten Docker Build

Context

Recently I started working on a microservices project, as a DevOps engineer my responsibility was to ensure smooth build and release of the project. One of the challenges that I was facing was the image building process of the projects was painfully slow. Following true Opstree spirit of continuous improvement I started exploring how I can fix this problem and finally got a decent success, I was able to reduce docker image build time from 4 minutes to 20 seconds. In this blog, I would like to showcase various ways through which image building can be reduced drastically.

Continue reading “Fasten Docker build”

Docker Networking – Containers Communication

In the modern world, the container is a fascinating technology, as it has revolutionized software development and delivery. Everyone is using containers because of its dynamic, scalable, and isolated nature.

People do use some orchestration software such as Kubernetes, Openshift, Docker Swarm, and AWS ECS, etc to run their production workloads on containers.

But the question is how these containers communicate with each other in various situations? Continue reading “Docker Networking – Containers Communication”

Create Your Own Container Using Linux Namespaces Part-1.

In this lock-down, everyone has to maintain a social distance and in this trying time, we can learn from docker to isolate ourselves. So before that, we need to learn how docker does it?
The best approach to learn is to simulate it. For that, we’ll be creating our own container tool for the application to isolate itself.

Self isolation | SpongeBob SquarePants | Know Your Meme

 

Continue reading “Create Your Own Container Using Linux Namespaces Part-1.”

Raktbeej Proxy

All we could think of imagining a routine day of a NOC guy is looking at all the fancy and colorful multiple screens around, but is this all it?

The answer to the above statement is NO! As a NOC, We have access to the information that is critical to analyze and plot company infra strength and on top of that, access to the servers and protected network makes the situation more critical if thing happens to be in wrong hands.

Continue reading “Raktbeej Proxy”

Docker Inside Out – A Journey to the Running Container

 

Necessity is the mother of invention, the same happens here in case of docker. With the pressure of splitting monolithic applications for the purpose of ease, we arrived at docker and it made our life much simpler. We all access docker with docker-cli command but I wonder what it does behind the scene, to run a container. Let’s get deeper into it in this very blog. Continue reading “Docker Inside Out – A Journey to the Running Container”

How to test Ansible playbook/role using Molecules with Docker

Why Molecule?

Have you ever faced issue that your Ansible code gets executed successfully but something went wrong like, service is not started, the configuration is not getting changed, etc?

Continue reading “How to test Ansible playbook/role using Molecules with Docker”

Docker Logging Driver

The  docker logs command batch-retrieves logs present at the time of execution. The docker logs command shows information logged by a running container. The docker service logs command shows information logged by all containers participating in a service. The information that is logged and the format of the log depends almost entirely on the container’s endpoint command.

These logs are basically stored at “/var/lib/docker/containers/.log”, So basically it is not easy to use this file by using Filebeat because the file will change every time when the new container is up with a new container id.

So, How to monitor these logs which are formed in different files ? For this Docker logging driver were introduced to monitor the docker logs.

Docker includes multiple logging mechanisms to help you get information from running containers & services. These mechanisms are called logging drivers. These logging drivers are configured for the docker daemon.

To configure the Docker daemon to default to a specific logging driver, set the value of log-driver to the name of the logging driver in the daemon.json file, which is located in /etc/docker/ on Linux hosts or C:\ProgramData\docker\config\ on Windows server hosts.

The default logging driver is json-file. The following example explicitly sets the default logging driver to syslog:

{                                            
  “log-driver”: “syslog”
}

After configuring the log driver in daemon.json file, you can define the log driver & the destination where you want to send the logs for example logstash & fluentd etc. You can define it either on the run time execution command as “–log-driver=syslog –log-opt syslog-address=udp://logstash:5044” or if you are using a docker-compose file then you can define it as:

“`
logging:
driver: fluentd
options:
fluentd-address: “192.168.1.1:24224”
tag: “{{ container_name }}”
“`

Once you have configured the log driver, it will send all the docker logs to the configured destination. And now if you will try to see the docker logs on the terminal using the docker logs command, you will get a msg:

“`
Error response from daemon: configured logging driver does not support reading
“`

Because all the logs have been parsed to the destination.

Let me give you an example that how i configured logging driver fluentd
and parse those logs onto Elasticsearch and viewed them on Kibana. In this case I am configuring the logging driver at the run-time by installing the logging driver plugin inside the fluentd but not in daemon.json. So make sure that your containers are created inside the same docker network where you will be configuring the logging driver.

Step 1: Create a docker network.

“`
docker network create docker-net
“`

Step 2: Create a container for elasticsearch inside a docker network.

“`
docker run -itd –name elasticsearch -p 9200:9200 –network=docker-net elasticsearch:6.4.1
“`

Step 3: Create a fluentd configuration where you will be configuring the logging driver inside the fluent.conf which is further being copied inside the fluentd docker image.

fluent.conf

“`

@type forward
port 24224
bind 0.0.0.0

@type copy

@type elasticsearch
host elasticsearch
port 9200
logstash_format true
logstash_prefix fluentd
logstash_dateformat %Y%m%d
include_tag_key true
type_name access_log
tag_key app
flush_interval 1s
index_name fluentd
type_name fluentd

@type stdout

“`

This will also create an index naming as fluentd & host is defined in the name of the service defined for elasticsearch.

Step 4: Build the fluentd image and create a docker container from that.

Dockerfile.fluent

“`
FROM fluent/fluentd:latest
COPY fluent.conf /fluentd/etc/
RUN [“gem”, “install”, “fluent-plugin-elasticsearch”, “–no-rdoc”, “–no-ri”, “–version”, “1.9.5”]
“`

Here the logging driver pluggin is been installed and configured inside the fluentd.

Now build the docker image. And create a container.

“`
docker build -t fluent -f Dockerfile.fluent .
docker run -itd –name fluentd -p 24224:24224 –network=docker-net fluent
“`

Step 5: Now you need to create a container whose logs you want to see on kibana by configuring it on the run time. In this example, I am creating an nginx container and configuring it for the log driver.

“`
docker run -itd –name nginx -p 80:80 –network=docker-net –log-driver=fluentd –log-opt fluentd-address=udp://:24224 opstree/nginx:server
“`

Step 6: Finally you need to create a docker container for kibana inside the same network.

“`
docker run -itd –name kibana -p 5601:5601 –network=docker-net kibana
“`

Now, You will be able to check the logs for the nginx container on kibana by creating an index fluentd-*.

Types of Logging driver which can be used:

       Driver           Description

  •  none:           No logs are available for the container and docker logs does  not return any output.
  •  json-file:     The logs are formatted as JSON. The default logging driver for Docker.
  •  syslog:     Writes logging messages to the syslog facility. The syslog daemon must be running on the host machine.
  •  journald:     Writes log messages to journald. The journald daemon must be running on the host machine.
  •  gelf:     Writes log messages to a Graylog Extended Log Format (GELF) endpoint such as Graylog or Logstash.
  •  fluentd:     Writes log messages to fluentd (forward input). The fluentd daemon must be running on the host machine.
  •  awslogs:     Writes log messages to Amazon CloudWatch Logs.
  •  splunk:     Writes log messages to splunk using the HTTP Event Collector.
  •  etwlogs:     Writes log messages as Event Tracing for Windows (ETW) events. Only available on Windows platforms.
  •  gcplogs:     Writes log messages to Google Cloud Platform (GCP) Logging.
  •  logentries:     Writes log messages to Rapid7 Logentries.