cgroups的实现

cgroups的实现

1. cgroups体系结构

每个内核的子系统如果想要挂载到cgroup系统中，必须要先拥有一个cgroup_subsys对象，通过这个对象将对子系统资源的操作函数以接口的形式进行约定，各个子系统再根据自己的需求去实现这些接口，除了这些函数指针，cgroup_subsys结构体还包括id, name, early_init等属性，分别表示该子系统在cgroup->subsys[]数组中的索引，子系统名称，在系统启动时是否需要尽早初始化等。

struct cgroup_subsys{
	struct cgroup_subsys_state* (*css_alloc)(struct cgroup_subsys_state *parent_css);
  void (*attach)(struct cgroup_taskset *tset);
  ...

  bool early_init: 1;
  ind id;
  const char *name;
  struct cgroup_root *root;
}

除了这些，cgroup_subsys还有一个指向cgroup_root类型的root指针，cgroup_root表示一个cgroup hierarchy的根，注意与根结点相区别，所以说一个子系统只能通过cgroup_root结构与一个cgroup hierarchy绑定（但是反过来一个cgroup hierarchy可以绑定多个子系统 ）。关于cgroup_root下面具体讨论。

再看第一个函数指针css_alloc，其返回了一个cgroup_subsys_state结构体，每一个注册到系统中的cgroup子系统，除了代表它的cgroup_subsys，还有与之相关联的cgroup_subsys_state，简称css，用来表示某个子系统与某个cgroup相关联的状态，将一个子系统与一个cgroup相连接。每个subsys会对调用css_alloc的cgroup分配一个自己的cgroup_subsys_state结构。

struct cgroup_subsys_state {
	/* PI: the cgroup that this css is attached to */
	struct cgroup *cgroup;

	/* PI: the cgroup subsystem that this css is attached to */
	struct cgroup_subsys *ss;

	/* reference count - access via css_[try]get() and css_put() */
	struct percpu_ref refcnt;

	/* siblings list anchored at the parent's ->children */
	struct list_head sibling;
	struct list_head children;

	/*
	 * PI: Subsys-unique ID.  0 is unused and root is always 1.  The
	 * matching css can be looked up using css_from_id().
	 */
	int id;

	unsigned int flags;
    ...

	struct cgroup_subsys_state *parent;
};

• cgroup指向此css所关联的cgroup
• ss指向此css所关联的cgroup_subsys
• sibling是一个双向链表，保存所关联的cgroup拥有的其他css
• parent指向父css的指针

提到css，那就不得不提css_set，css_set是保存指向cgroup_subsys_state对象的一组引用计数指针的集合，用来保存与task相关的cgroup信息，同时也代表了一组子系统资源的组合，css_set的使用节省了cgroups在task_struct结构体中所占用的空间，而且加快了task fork()/exit()的速度。

struct css_set {
	/*
	 * Set of subsystem states, one for each subsystem. This array is
	 * immutable after creation apart from the init_css_set during
	 * subsystem registration (at boot time).
	 */
	struct cgroup_subsys_state *subsys[CGROUP_SUBSYS_COUNT];

	/* reference count */
	refcount_t refcount;
  ...
	/*
	 * Lists running through all tasks using this cgroup group.
	 */
	struct list_head tasks;

	/*
	 * List running through all cgroup groups in the same hash
	 * slot. Protected by css_set_lock
	 */
	struct hlist_node hlist;

	/*
	 * List of cgrp_cset_links pointing at cgroups referenced from this
	 * css_set.  Protected by css_set_lock.
	 */
	struct list_head cgrp_links;

	/* dead and being drained, ignore for migration */
	bool dead;

	/* For RCU-protected deletion */
	struct rcu_head rcu_head;
};

• subsys保存子系统状态的集合，初始化后不可修改
• hlist用来链接同一个hashtable slots中的所有css_set
• refcount用来引用计数
• task用来链接所有使用此css_set的task_struct集合

从task到cgroup的关联

从task到其所属的cgroup之间是没有直接指针相连接的，但是task可以通过一个媒介来获取其所属的cgroup，这个媒介就是css_set和cgroup_subsys_state。通过task_struct -> cgroups -> subsys[ssid] ->cgroup即可访问到管理对应子系统的cgroup。之所以这么设计时因为获取子系统状态的操作预计会频繁发生，而且是在性能关键代码中，然而需要一个task实际的cgroup来执行的操作（尤其是task在cgroups之间迁移的操作）则并没有那么常见。task_struct中的cg_list则是用来连接使用同一个css_set的task的链表，css_set通过tasks来遍历访问此链表。

css_set与cgroup之间的关联

一个进程属于一个css_set, 一个css_set就存储了一组进程跟各个子系统相关的信息，但是这些信息有可能不是从一个cgroup那里获得的，因为一个进程可以同时属于几个cgroup，只要这些cgroup不在同一个层级。所以一个css_set存储的cgroup_subsys_state可以对应多个cgroup。

另一方面，cgroup也存储了一组cgroup_subsys_state，这一组cgroup_subsys_state则是cgroup从所在的层级附加的子系统获得的。一个cgroup中可以有多个进程，而这些进程的css_set不一定都相同，因为有些进程可能还加入了其他cgroup。但是同一个cgroup中的进程与该cgroup关联的cgroup_subsys_state都受到该cgroup的管理(cgroups中进程控制是以cgroup为单位的)的，所以一个cgroup也可以对应多个css_set。

因此它们之间的关系是 $M\ast N$ , 这种关系的表达则是通过cgrp_cset_link来实现

struct cgrp_cset_link {
	/* the cgroup and css_set this link associates */
	struct cgroup		*cgrp;
	struct css_set		*cset;

	/* list of cgrp_cset_links anchored at cgrp->cset_links */
	struct list_head	cset_link;

	/* list of cgrp_cset_links anchored at css_set->cgrp_links */
	struct list_head	cgrp_link;
};

在回到我们一开始所讲的cgroup_subsys结构，其拥有的一个属性名为root，指向一个cgroup_root结构体

struct cgroup_root {
	struct kernfs_root *kf_root;

	/* The bitmask of subsystems attached to this hierarchy */
	unsigned int subsys_mask;

	/* Unique id for this hierarchy. */
	int hierarchy_id;

	/* The root cgroup. Root is destroyed on its release. */
	struct cgroup cgrp;

	/* for cgrp->ancestor_ids[0] */
	u64 cgrp_ancestor_id_storage;

	/* Number of cgroups in the hierarchy, used only for /proc/cgroups */
	atomic_t nr_cgrps;

	/* A list running through the active hierarchies */
	struct list_head root_list;

	/* Hierarchy-specific flags */
	unsigned int flags;

	/* The path to use for release notifications. */
	char release_agent_path[PATH_MAX];

	/* The name for this hierarchy - may be empty */
	char name[MAX_CGROUP_ROOT_NAMELEN];
};

一个cgroup_root是一个层级的根，是cgroup的核心，且不由controller直接操作。

• kf_root：
• subsys_mask：与该层级相关联的子系统的点位图
• hierarchy_id：层级的ID
• cgrp：层级的根节点
• root_list：包含所有层级的链表

接下来引入主角cgroup

struct cgroup {
	/* self css with NULL ->ss, points back to this cgroup */
	struct cgroup_subsys_state self;

	unsigned long flags;		/* "unsigned long" so bitops work */

	/*
	 * The depth this cgroup is at.  The root is at depth zero and each
	 * step down the hierarchy increments the level.  This along with
	 * ancestor_ids[] can determine whether a given cgroup is a
	 * descendant of another without traversing the hierarchy.
	 */
	int level;

	/* Maximum allowed descent tree depth */
	int max_depth;
	...


	struct kernfs_node *kn;		/* cgroup kernfs entry */
	struct cgroup_file procs_file;	/* handle for "cgroup.procs" */
	struct cgroup_file events_file;	/* handle for "cgroup.events" */


	/* Private pointers for each registered subsystem */
	struct cgroup_subsys_state __rcu *subsys[CGROUP_SUBSYS_COUNT];

	struct cgroup_root *root;

	/*
	 * List of cgrp_cset_links pointing at css_sets with tasks in this
	 * cgroup.  Protected by css_set_lock.
	 */
	struct list_head cset_links;


	/* per-cpu recursive resource statistics */
	struct cgroup_rstat_cpu __percpu *rstat_cpu;
	struct list_head rstat_css_list;

	/* cgroup basic resource statistics */
	struct cgroup_base_stat last_bstat;
	struct cgroup_base_stat bstat;
	struct prev_cputime prev_cputime;	/* for printing out cputime */

	/*
	 * list of pidlists, up to two for each namespace (one for procs, one
	 * for tasks); created on demand.
	 */
	struct list_head pidlists;
	struct mutex pidlist_mutex;

	/* used to wait for offlining of csses */
	wait_queue_head_t offline_waitq;

	/* used to schedule release agent */
	struct work_struct release_agent_work;

	/* If there is block congestion on this cgroup. */
	atomic_t congestion_count;

	/* Used to store internal freezer state */
	struct cgroup_freezer_state freezer;

	/* ids of the ancestors at each level including self */
	u64 ancestor_ids[];
};

• sibling, children, parent三个list_head负责将同一层级的cgroup连接为一颗cgroup树
• subsys存储一组指向cgroup_subsys_state的指针
• root指向层级所对应的cgroup_root结构体
• cset_links指向cgrp_cset_link连成的链表，负责与css_set相连接

2. cgroup文件系统

cgroup v1,v2的文件系统类型的数据结构

struct file_system_type cgroup_fs_type = {
	.name = "cgroup",
	.mount = cgroup_mount,
	.kill_sb = cgroup_kill_sb,
	.fs_flags = FS_USERNS_MOUNT,
};

static struct file_system_type cgroup2_fs_type = {
	.name = "cgroup2",
	.mount = cgroup_mount,
	.kill_sb = cgroup_kill_sb,
	.fs_flags = FS_USERNS_MOUNT,
};

cgroup文件系统其实是通过kernfs来实现的，通过文件接口的形式将内核中cgroup信息以及子系统信息传递给用户态，用户照样通过调用vfs接口来进行一般的读写操作。

以cgroup创建为例，内核创建一个cgroup首先调用cgroup_mkdir，cgroup_mkdir调用cgroup_create和css_populate_dir，cgroup_create调用kernfs_create_dir在cgroup文件系统中创建cgroup对应的目录，css_populate_dir又调用cgroup_addrm_dir在对应cgroup目录中填充对应子系统控制文件。

那么是如果通过向控制文件中写入或删除字符（串）来实现资源控制呢？答案就在struct cftype中，cftype在源码中被定义为handler for definitions for cgroup control files。每个子系统通过实现自己的cftype数组（每个控制文件都对应一个cftype结构体，所以每个cgroup_subsys中都保存了一个cftype数组）来实现各自的资源分配功能。

其具体定义如下：

struct cftype {
	/*
	 * By convention, the name should begin with the name of the
	 * subsystem, followed by a period.  Zero length string indicates
	 * end of cftype array.
	 */
	char name[MAX_CFTYPE_NAME];
	unsigned long private;

	/*
	 * The maximum length of string, excluding trailing nul, that can
	 * be passed to write.  If < PAGE_SIZE-1, PAGE_SIZE-1 is assumed.
	 */
	size_t max_write_len;

	/* CFTYPE_* flags */
	unsigned int flags;

	/*
	 * If non-zero, should contain the offset from the start of css to
	 * a struct cgroup_file field.  cgroup will record the handle of
	 * the created file into it.  The recorded handle can be used as
	 * long as the containing css remains accessible.
	 */
	unsigned int file_offset;

	/*
	 * Fields used for internal bookkeeping.  Initialized automatically
	 * during registration.
	 */
	struct cgroup_subsys *ss;	/* NULL for cgroup core files */
	struct list_head node;		/* anchored at ss->cfts */
	struct kernfs_ops *kf_ops;

	int (*open)(struct kernfs_open_file *of);
	void (*release)(struct kernfs_open_file *of);

	/*
	 * read_u64() is a shortcut for the common case of returning a
	 * single integer. Use it in place of read()
	 */
	u64 (*read_u64)(struct cgroup_subsys_state *css, struct cftype *cft);
	/*
	 * read_s64() is a signed version of read_u64()
	 */
	s64 (*read_s64)(struct cgroup_subsys_state *css, struct cftype *cft);

	/* generic seq_file read interface */
	int (*seq_show)(struct seq_file *sf, void *v);

	/* optional ops, implement all or none */
	void *(*seq_start)(struct seq_file *sf, loff_t *ppos);
	void *(*seq_next)(struct seq_file *sf, void *v, loff_t *ppos);
	void (*seq_stop)(struct seq_file *sf, void *v);

	/*
	 * write_u64() is a shortcut for the common case of accepting
	 * a single integer (as parsed by simple_strtoull) from
	 * userspace. Use in place of write(); return 0 or error.
	 */
	int (*write_u64)(struct cgroup_subsys_state *css, struct cftype *cft,
			 u64 val);
	/*
	 * write_s64() is a signed version of write_u64()
	 */
	int (*write_s64)(struct cgroup_subsys_state *css, struct cftype *cft,
			 s64 val);

	/*
	 * write() is the generic write callback which maps directly to
	 * kernfs write operation and overrides all other operations.
	 * Maximum write size is determined by ->max_write_len.  Use
	 * of_css/cft() to access the associated css and cft.
	 */
	ssize_t (*write)(struct kernfs_open_file *of,
			 char *buf, size_t nbytes, loff_t off);

	__poll_t (*poll)(struct kernfs_open_file *of,
			 struct poll_table_struct *pt);

#ifdef CONFIG_DEBUG_LOCK_ALLOC
	struct lock_class_key	lockdep_key;
#endif
};

可以看到cfytpe定义了很多文件的函数指针，目的就是将用户对控制文件输入的字符（串）转换为对进程的资源管控，每一个cgroup 目录下的控制文件对应的cftype都包含cgroup_base_files + css -> ss -> cfts，分别对应来自cgroup和来自子系统的控制文件，cgroup的控制文件一般以cgroup.开头，来自子系统的控制文件一般以子系统的名称开头。

cgroup通过调用cgroup_addrm_file来添加删除控制文件，并将cftype保存到对应kernfs文件

cftype flags的类型

/* cftype->flags */
enum {
	CFTYPE_ONLY_ON_ROOT	= (1 << 0),	/* only create on root cgrp */
	CFTYPE_NOT_ON_ROOT	= (1 << 1),	/* don't create on root cgrp */
	CFTYPE_NS_DELEGATABLE	= (1 << 2),	/* writeable beyond delegation boundaries */

	CFTYPE_NO_PREFIX	= (1 << 3),	/* (DON'T USE FOR NEW FILES) no subsys prefix */
	CFTYPE_WORLD_WRITABLE	= (1 << 4),	/* (DON'T USE FOR NEW FILES) S_IWUGO */
	CFTYPE_DEBUG		= (1 << 5),	/* create when cgroup_debug */

	/* internal flags, do not use outside cgroup core proper */
	__CFTYPE_ONLY_ON_DFL	= (1 << 16),	/* only on default hierarchy */
	__CFTYPE_NOT_ON_DFL	= (1 << 17),	/* not on default hierarchy */
};