Performance Comparison Between Singularity And Docker

Abstract—To meet the growing demand for computing power, HPC clusters have grown tremendously in size and complexity. With the prevalence of high-speed interconnects, multi-core processors, and gas pedals, sharing these resources efficiently has become even more important to achieve faster turnaround times and lower costs per user. the HPC cluster environment invokes a large number of analytic programs as well as internal development scripts, and the environment is extremely complex to configure and manage, with low repeatability, leading to big challenges in upgrading, managing, and migrating processes. Existing IT technologies have a solution for this: containers. However, different containers have different performance in various aspects. This article compares the performance differences between docker and singularity in a comprehensive way, with the aim of getting a condition of the applicability of these two containers in different scenarios.

I. INTRODUCTION

Over time, after the technology of hardware virtualization, abundant hardware resources have become more accessible and easy to use than ever before, and this has led to what we know today as cloud platforms. Most of the cloud services are based on virtual machines. The performance of the virtual machine has a significant impact on the overall performance of the cloud application. For tasks that require computing, this is the biggest drawback of the technology. Because different operating systems use different kernels, such as Windows and Linux, any kernel used in a system consumes a large amount of computing resources such as CPU and memory, which are the resources most needed for high-performance computing. The goal of containerization is to provide more computational power to applications by sacrificing this flexibility, so all containers use the same cores. Due to this limitation, containers cannot be migrated live and must be suspended or stopped before they can be moved to another host. On top of that, most container scenarios use static containers to secure business applications, and these containers are rarely moved at runtime. Containers use the host's working kernel when they are working.

A container is a lightweight operating system within the host OS that runs them. It uses the native instructions of the core CPU and does not has any VMM (Virtual Machine Manager) requirements. The only limitation is that containers must use the kernel of the host operating system to use existing hardware components. Unlike virtualization, containers can use different operating systems without any restrictions, and we would like to know if there are unacceptable performance overheads or security issues when containers run on different operating systems.

Traditional hypervisor-based virtualization solutions are not commonly used in high performance computing (HPC) due to the performance overhead. Container-based virtualization technologies, represented by Linux Container and Docker, can provide better resource sharing, customized environments and low overhead. We can see that different containers have different advantages and disadvantages. For example, Docker is definitely the hottest container technology today and does a great job in terms of version control, isolation, portability, etc. However, it has some permission issues when used by non-root users and includes unnecessary resource overhead, perhaps making it not well suited to the functional and security needs in HPC environments. Singularity is an attractive container-based approach to meet the requirements of scientific applications. It features computational mobility, repeatability, user freedom, and support for existing traditional HPC, but it does not provide very complete isolation and virtualization as Docker does.

Although some studies have shown that docker outperforms virtualization solutions in most aspects and has comparable performance with bare metal, there are also studies that have tested the advantages of singularity in hpc environments such as lightweight, low overhead, and adaptability to hpc environments. We already know enough that docker and singularity have excellent performance, but at the same time we have to admit that these containers have shortcomings and there are relatively good choices in different application scenarios. But according to our investigation we did not find these two excellent containers together for full testing and comparison. So the purpose of this article is not only to focus on their performance evaluation with benchmarks, but also to investigate their practicality in many aspects to come up with targeted application criteria, which is also our motivation.

Ⅱ. Contribution

We summarized our work process and conclusions into a blog. First, we surveyed and summarized the relevant work, then briefly introduced several containers and some indicators to judge the practicability of containers, and explained the installation and deployment process of docker and singularity. Then is our experimental test. We have carried out a unified test on the machine in the laboratory. Through the benchmark test, we have measured the throughput, latency, cpu utilization and other performance of the container on single thread and multiple threads, and have extended the comparison to multiple nodes in the recent period. Through consulting the documents, the performance of version control, isolation, portability, repeatability and other performances are compared, and finally the container use recommendations adapted to different situations are obtained.

Ⅲ. RELATED WORK

Xavier et al. [1] conducted some experiments to evaluate container performance for high performance computing (HPC). They compared OpenVZ, LXC, VServer, and Xen, considering isolation issues. They conclude that all container-based systems have almost native performance in terms of CPU, memory, disk and network. The main difference between them is the implementation of resource management, leading to problems in isolation and security.

Beserra et al. [2] analyzed the performance of LXC containers in comparison to hypervisor-based virtualization - KVM - for HPC activities. The results showed that the type of hypervisor directly affects the results. They conclude that LXC is better suited for HPC than KVM, however, in more complex cases where physical resources are partitioned into multiple and logical environments, the performance of both degrades, especially for KVM. LXC's superior performance is also observed in clustered environments where there is more cooperation between processes. Some issues have been reported with LXC compared to KVM, such as isolation issues.

Pedretti et al. [3] test Shifter and Singularity in HPC systems. tested and compared bare-metal execution and container execution in supercomputers (NERSC's Edison and Sandia's Volta). Their experiments included synthetic benchmarks and computational tools such as HPCG, HPMG-FE, IMB, and FEniCS toolboxes. Similary, Ruiz et al. [4] investigate the impact of using containers in the context of HPC research. The evaluation showed the limitations of using containers, which types of applications are most affected, and what level of oversubscription containers can handle without affecting application performance. While using containers gains considerable overhead, it shows that the technology is becoming more mature and that performance issues are being addressed with each new release of the Linux kernel.

A comparative study of productive simulations using biological systems is provided by Rudyy et al. [5]. In the paper, the prod