在容器中使用CGroups

在容器中使用CGroups

我们在 Containerd 的容器中来使用 cgroup，比如使用 docker 启动一个 nginx 容器，并限制其使用内存为50M:

[root@master1 ~]# docker  run -d -m 50m --name nginx nginx:1.7.9
7c53de927a8013e2454e80ff1c56621a28578b4b471d5a9ef7ab0514062671e9
[root@master1 ~]# docker ps |grep nginx
7c53de927a80   nginx:1.7.9                                         "nginx -g 'daemon of…"   25 seconds ago   Up 24 seconds   80/tcp, 443/tcp   nginx
7dd9f07a43ce   nginx                                               "/docker-entrypoint.…"   2 days ago       Up 2 days       80/tcp            webserver

在使用docker run 启动容器的时候可以使用 -m 或 --memory 参数来现在内存，启动完成后该容器的 cgroup 会出现在分散在各个cgroup子系统的system.slice目录下面；

Ps：按照官方说明，启动完成后该容器的 cgroup 会出现在 名为 default 的目录下面，待探究。

那么这里与标准结果不一样，我通过以下步骤验证操作容器是否由cgroup控制组控制的：

1. 通过docker inspect 查看容器id
2. 在cgroup目录下查询是否存在

[root@master1 cgroup]# docker inspect nginx ##也可以grep过滤
[
    {
        "Id": "7c53de927a8013e2454e80ff1c56621a28578b4b471d5a9ef7ab0514062671e9",
        "Created": "2023-02-05T05:08:02.153805812Z",
        "Path": "nginx",
        "Args": [
            "-g",
            "daemon off;"
        ],
...

[root@master1 cgroup]# pwd
/sys/fs/cgroup

[root@master1 cgroup]# find ./ -name "*7c53de927a8013e2454e80ff1c56621a28578b4b471d5a9ef7ab0514062671e9*"
./perf_event/system.slice/docker-7c53de927a8013e2454e80ff1c56621a28578b4b471d5a9ef7ab0514062671e9.scope
./pids/system.slice/docker-7c53de927a8013e2454e80ff1c56621a28578b4b471d5a9ef7ab0514062671e9.scope
./blkio/system.slice/docker-7c53de927a8013e2454e80ff1c56621a28578b4b471d5a9ef7ab0514062671e9.scope
./freezer/system.slice/docker-7c53de927a8013e2454e80ff1c56621a28578b4b471d5a9ef7ab0514062671e9.scope
./devices/system.slice/docker-7c53de927a8013e2454e80ff1c56621a28578b4b471d5a9ef7ab0514062671e9.scope
./hugetlb/system.slice/docker-7c53de927a8013e2454e80ff1c56621a28578b4b471d5a9ef7ab0514062671e9.scope
./cpu,cpuacct/system.slice/docker-7c53de927a8013e2454e80ff1c56621a28578b4b471d5a9ef7ab0514062671e9.scope
./cpuset/system.slice/docker-7c53de927a8013e2454e80ff1c56621a28578b4b471d5a9ef7ab0514062671e9.scope
./net_cls,net_prio/system.slice/docker-7c53de927a8013e2454e80ff1c56621a28578b4b471d5a9ef7ab0514062671e9.scope
./memory/system.slice/docker-7c53de927a8013e2454e80ff1c56621a28578b4b471d5a9ef7ab0514062671e9.scope
./systemd/system.slice/docker-7c53de927a8013e2454e80ff1c56621a28578b4b471d5a9ef7ab0514062671e9.scope

上面我们启动的 nginx 容器 ID 的目录会出现在 /sys/fs/cgroup/memory/system.slice下面，该文件夹下面有很多和内存相关的 cgroup 配置文件，要进行相关的配置就需要在该目录下对应的文件中去操作：

[root@master1 system.slice]# ll docker-7c53de927a8013e2454e80ff1c56621a28578b4b471d5a9ef7ab0514062671e9.scope/
total 0
-rw-r--r-- 1 root root 0 Feb  5 00:08 cgroup.clone_children
--w--w--w- 1 root root 0 Feb  5 00:08 cgroup.event_control
-rw-r--r-- 1 root root 0 Feb  5 00:08 cgroup.procs
-rw-r--r-- 1 root root 0 Feb  5 00:08 memory.failcnt
--w------- 1 root root 0 Feb  5 00:08 memory.force_empty
-rw-r--r-- 1 root root 0 Feb  5 00:08 memory.kmem.failcnt
-rw-r--r-- 1 root root 0 Feb  5 00:08 memory.kmem.limit_in_bytes
-rw-r--r-- 1 root root 0 Feb  5 00:08 memory.kmem.max_usage_in_bytes
-r--r--r-- 1 root root 0 Feb  5 00:08 memory.kmem.slabinfo
-rw-r--r-- 1 root root 0 Feb  5 00:08 memory.kmem.tcp.failcnt
-rw-r--r-- 1 root root 0 Feb  5 00:08 memory.kmem.tcp.limit_in_bytes
-rw-r--r-- 1 root root 0 Feb  5 00:08 memory.kmem.tcp.max_usage_in_bytes
-r--r--r-- 1 root root 0 Feb  5 00:08 memory.kmem.tcp.usage_in_bytes
-r--r--r-- 1 root root 0 Feb  5 00:08 memory.kmem.usage_in_bytes
-rw-r--r-- 1 root root 0 Feb  5 00:08 memory.limit_in_bytes
-rw-r--r-- 1 root root 0 Feb  5 00:08 memory.max_usage_in_bytes
-rw-r--r-- 1 root root 0 Feb  5 00:08 memory.memsw.failcnt
-rw-r--r-- 1 root root 0 Feb  5 00:08 memory.memsw.limit_in_bytes
-rw-r--r-- 1 root root 0 Feb  5 00:08 memory.memsw.max_usage_in_bytes
-r--r--r-- 1 root root 0 Feb  5 00:08 memory.memsw.usage_in_bytes
-rw-r--r-- 1 root root 0 Feb  5 00:08 memory.move_charge_at_immigrate
-r--r--r-- 1 root root 0 Feb  5 00:08 memory.numa_stat
-rw-r--r-- 1 root root 0 Feb  5 00:08 memory.oom_control
---------- 1 root root 0 Feb  5 00:08 memory.pressure_level
-rw-r--r-- 1 root root 0 Feb  5 00:08 memory.soft_limit_in_bytes
-r--r--r-- 1 root root 0 Feb  5 00:08 memory.stat
-rw-r--r-- 1 root root 0 Feb  5 00:08 memory.swappiness
-r--r--r-- 1 root root 0 Feb  5 00:08 memory.usage_in_bytes
-rw-r--r-- 1 root root 0 Feb  5 00:08 memory.use_hierarchy
-rw-r--r-- 1 root root 0 Feb  5 00:08 notify_on_release
-rw-r--r-- 1 root root 0 Feb  5 00:08 tasks

我们这里需要关心的是 memory.limit_in_bytes 文件，该文件就是用来设置内存大小的，正常应该是 50M 的内存限制：

[root@master1 docker-7c53de927a8013e2454e80ff1c56621a28578b4b471d5a9ef7ab0514062671e9.scope]# cat memory.limit_in_bytes 
52428800
[root@master1 docker-7c53de927a8013e2454e80ff1c56621a28578b4b471d5a9ef7ab0514062671e9.scope]# cat ../memory.limit_in_bytes 
9223372036854771712

[root@master1 docker-7c53de927a8013e2454e80ff1c56621a28578b4b471d5a9ef7ab0514062671e9.scope]# bc ##辅助计算
bc 1.06.95
Copyright 1991-1994, 1997, 1998, 2000, 2004, 2006 Free Software Foundation, Inc.
This is free software with ABSOLUTELY NO WARRANTY.
For details type `warranty'. 

52428800/1024/1024（换算成50M）
50

我们看到本机目录下内存限制与我们创建容器时-m 50m 一致，而上级目录下内存限制仍然是默认的大小

同样我们的 nginx 容器进程 ID 也会出现在上面的 tasks 文件中：

[root@master1 docker-7c53de927a8013e2454e80ff1c56621a28578b4b471d5a9ef7ab0514062671e9.scope]# cat tasks 
29073
29107

我们可以通过如下命令过滤该进程号，可以看出第一行的 29073 就是 nginx 进程在主机上的进程 ID，下面的是这个进程下的线程：

[root@master1 docker-7c53de927a8013e2454e80ff1c56621a28578b4b471d5a9ef7ab0514062671e9.scope]# ps aux|grep nginx
root     29073  0.0  0.0  31064  2780 ?        Ss   00:08   0:00 nginx: master process nginx -g daemon off;
101      29107  0.0  0.0  31440  1656 ?        S    00:08   0:00 nginx: worker process

我们删除/停止这个容器后，/sys/fs/cgroup/相关目录下的容器 ID 文件夹也会自动删除。