Another Reason Why Your Docker Containers May be Slow

Docker is now all over the place. Over the past few years, software has now moved to be a package in a Docker container. One of the biggest benefits which you got from Docker is its speed. You would not get lightning-fast performance out of the box without the Docker performance tuning.

Most of the organizations today are working under Docker. They are running thousands of automated tests every day. After containerize the tests applications, you may notice that some parts of the software ran more slowly in Docker rather than any other resources. It has been seen that some extra time has been spent somewhere in native code. However, a proper Docker Online training is needed to know about the details of issues regarding slow of Docker Containers.

A Research of a Company for Having Slow of Docker Containers:

According to an article of a big company, while they were using their internal Kubernetes cluster to run a bunch of Dev-relate a CI/CD workflows and everything was running smoothly except one thing - when launching Dockerized copies of the product, they saw very worse performance than they expected. Even the CPU and memory limits are of 5 CPU/30 GB RAM set where the containers set through the Pod configuration. On the virtual machine, it could be more than enough for all the queries upon the tiny test dataset to work. But in case of Docker, they are able to launch only 3 to 4 copies products on a 72CPU/512 Gb RAM Machine before thing was running too slow. The queries which may take only milliseconds to complete but now taking a second or two which is ultimately causing failures in the CI pipelines. So, they started researching it.

Suspected Reasons of Having Slow Docker Containers:

• The primary suspect was definitely the configuration errors that might occur during packaging the product in Docker. However, there was not so much issue which can cause slowness as compare to VM or bare-metal installations. 

• The next step was the running of all types of tests from a Sysbench Package. They started testing RAM, CPU, disk performance but nothing looked any different from bare metal. Even some of the services of the product have been save detailed traces of all the activities so that it can be used later for performance profiling. 

• Even if there will be any starving on one of the resources (CPU, RAM, Network, disk) there would be similar skews in timing for some calls so that it can determine where the slowness comes from. But in this case, nothing was looked wrong. All-time, the proportion was same as in the healthy configuration except that every call was considerably slower than on bare metal. 

Reason behind Your Slow Docker Containers

When nothing was detected, at that moment they found another resource where similar problem arises with Docker Container is going slow. Here, two supposedly lightweight, processes were killing each other while running Docker on the same machine even after resource limits were set to very conservative values. The two keys that are creating issues are –

1. The root cause the problem was ended up being in the Linux Kernel. Due to a Kernel dentry cache design, the behaviour of one process was making _d_lookup_loop kernel call considerably slower and this was just affecting the performance and making the Docker performance slower.

2. Using some debugging tools like perf to track down a kernel bug is really a great idea. It is a performance analyzing tool in Linux that is available in Linux Kernel version 2.6.31.

Now, after using perf, the work went easier. They noticed, the top 5 calls are kernel-related and the time is mostly spending on the kernel space and in other cases, most of the time is spent by their own processes operating in the userspace. A call that is taking all the time is posix_fadvise. With proper Docker Training in Hyderabad or any other places of India or through online, one can get all the suspected and original reasons of slow down of your Docker Containers and several tips and tricks of improving the speed.