jail管理器CBSD实践@FreeBSD

CBSD介绍

CBSD是为FreeBSD jail子系统、bhyve、QEMU/NVMM和Xen编写的管理层。该项目定位为一个综合解决方案的单一集成工具，用于使用预定义的软件集以最少的配置快速构建和部署计算机虚拟环境。
虽然CBSD没有提供额外的操作系统级功能，但它极大地简化了原本需要用户通过命令行界面（CLI）手动执行数十甚至数百个命令的复杂过程。CBSD的存在正是为了解决这一繁琐任务，让人能够更高效、更便捷地管理您的虚拟化环境，而非陷入无休止的命令行操作中。

官网：CBSD — FreeBSD Jail and Bhyve Management Tools

视频：FOSDEM 2021 - Managing virtual resources with CBSD, and beyond 

管理cbsd ubuntu jail不能使用默认的cbsd jexec jailname ,而是要用 jexec  jailname sh ,具体见：FreeBSD通过CBSD管理低资源容器jail来安装Ubuntu子系统实践-CSDN博客

CBSD安装

pkg安装

pkg install cbsd

CBSD初始化

如果是使用zfs系统，需要先执行：

/sbin/zfs create -o mountpoint=/usr/jails -o atime=off zroot/jails

 然后进行初始化即可：

env workdir=/usr/jails /usr/local/cbsd/sudoexec/initenv

按照提示一步步作答就行，下面是具体操作流程：

env workdir=/usr/jails /usr/local/cbsd/sudoexec/initenv
-------[CBSD v.14.0.7]-------
 This is install/upgrade scripts for CBSD.
 Don't forget to backup.
-----------------------------
Do you want prepare or upgrade hier environment for CBSD now?
[yes(1) or no(0)]
y
>>> Installing or upgrading
[Stage 1: account & dir hier]

Error: on ZFS-based systems, the CBSD requires a separate dataset different from the root one (zroot/ROOT/default).
Please create a separate dataset first, e.g.:

 /sbin/zfs create -o mountpoint=/usr/jails -o atime=off zroot/jails

Then re-run: env workdir=/usr/jails /usr/local/cbsd/sudoexec/initenv
root@fbhost:~ # 
root@fbhost:~ # /sbin/zfs create -o mountpoint=/usr/jails -o atime=off zroot/jails
root@fbhost:~ # env workdir=/usr/jails /usr/local/cbsd/sudoexec/initenv
-------[CBSD v.14.0.7]-------
 This is install/upgrade scripts for CBSD.
 Don't forget to backup.
-----------------------------
Do you want prepare or upgrade hier environment for CBSD now?
[yes(1) or no(0)]
y
>>> Installing or upgrading
[Stage 1: account & dir hier]
 * Check hier and permission...
 * write directory id: jaildatadir
 * write directory id: jailsysdir
 * write directory id: jailrcconfdir
 * write directory id: dbdir
[Stage 2: build tools]
Shall I add the cbsd user into /usr/local/etc/sudoers.d sudo file to obtain root privileges for most of the cbsd commands?
[yes(1) or no(0)]
yes
cmp: /usr/local/etc/sudoers.d: Is a directory
[Stage 3: local settings]
Shall i modify the /etc/rc.conf to sets cbsd_workdir="/usr/jails"?: 
[yes(1) or no(0)]
y
/etc/rc.conf: cbsd_workdir:  -> /usr/jails
[Stage 4: update default skel resolv.conf]
[Stage 5: refreshing inventory]
nodename: CBSD Nodename for this host e.g. the hostname. Warning: this operation will recreate the ssh keys in /usr/jails/.ssh dir: fbhost

Empty inventory database created: /usr/jails/var/db/inv.fbhost.sqlite

nodeip: Node management IPv4 address (used for node interconnection), e.g: 192.168.1.5
jnameserver: environment default DNS name-server (for jails resolv.conf), e.g.: 9.9.9.9,149.112.112.112,2620:fe::fe,2620:fe::9
114.114.114.114
nodeippool:  (networks for jails)
Hint: use space as delimiter for multiple networks, e.g.: 10.0.0.0/16

nat_enable: Enable NAT for RFC1918 networks?
[yes(1) or no(0)]
y
Which NAT framework do you want to use: [pf]
(type FW name, eg.: pf,ipfw,ipfilter, 'disable' or '0' to CBSD NAT, "exit" for break)
pf
Set IP address or NIC as the aliasing NAT address or interface, e.g: 192.168.1.5

Do you want to modify /boot/loader.conf to set pf_load=YES ?
[yes(1) or no(0)]
1
/boot/loader.conf: pf_load:  -> YES
fbsdrepo: Use official FreeBSD repository? When no (0) the repository of CBSD is preferred (useful for stable=1) for fetching base/kernel?
[yes(1) or no(0)]
1
zfsfeat: You are running on a ZFS-based system. Enable ZFS feature?
[yes(1) or no(0)]
1
parallel: Parallel mode stop/start ?
(0 - no parallel or positive value (in seconds) as timeout for next parallel sequence) e.g: 5

stable: Use STABLE branch instead of RELEASE by default? Attention: only the CBSD repository has a binary base for STABLE branch ?
(STABLE_X instead of RELEASE_X_Y branch for base/kernel will be used), e.g.: 0 (use release)

sqlreplica: Enable sqlite3 replication to remote nodes ?
(0 - no replica, 1 - try to replicate all local events to remote nodes) e.g: 1

statsd_bhyve_enable: Configure CBSD statsd services for collect RACCT bhyve statistics? ?
(EXPERIMENTAL FEATURE)? e.g: 0

statsd_jail_enable: Configure CBSD statsd services for collect RACCT jail statistics? ?
(EXPERIMENTAL FEATURE)? e.g: 0

statsd_hoster_enable: Configure CBSD statsd services for collect RACCT hoster statistics? ?
(EXPERIMENTAL FEATURE)? e.g: 0

[Stage 6: authentication keys]
Generating public/private ed25519 key pair.
Your identification has been saved in /usr/jails/.ssh/8536245b885a14e578e1094179cf14cc.id_rsa
Your public key has been saved in /usr/jails/.ssh/8536245b885a14e578e1094179cf14cc.id_rsa.pub
The key fingerprint is:
SHA256:2BoIPxFGCAOWNx/+WJuF31KFhe1G3PP5JOjTU0VBfb4 root@fbhost
The key's randomart image is:
+--[ED25519 256]--+
|=o.o+       *.o++|
|.o.+ o     o = o+|
|  o = . .   +. .*|
|   o = = . ..o..=|
|    + * S o... +o|
|     o * o .o oE.|
|      .   .  . . |
|                 |
|                 |
+----[SHA256]-----+
[Stage 7: nodes]
[Stage 8: modules]
Installing module pkg.d cmd: pkg
Installing module bsdconf.d cmd: tzsetup
Installing module bsdconf.d cmd: ssh
Installing module bsdconf.d cmd: ftp
Installing module bsdconf.d cmd: adduser
Installing module bsdconf.d cmd: passwd
Installing module bsdconf.d cmd: service
Installing module bsdconf.d cmd: sysrc
Installing module bsdconf.d cmd: userlist
Installing module bsdconf.d cmd: grouplist
Installing module bsdconf.d cmd: adduser-tui
Installing module bsdconf.d cmd: pw
Installing module bsdconf.d cmd: cloudinit
Installing module zfsinstall.d cmd: zfsinstall
[Stage 9: cleanup]
 * Remove obsolete files...
Configure RSYNC services for jail migration?
[yes(1) or no(0)]
1
Shall I modify /etc/rc.conf to set cbsdrsyncd_enable="YES"
[yes(1) or no(0)]
1
/etc/rc.conf: cbsdrsyncd_enable:  -> YES
Do you want to modify /etc/rc.conf to set the cbsdrsyncd_flags="--config=/usr/jails/etc/rsyncd.conf" ?
[yes(1) or no(0)]
1
/etc/rc.conf: cbsdrsyncd_flags:  -> --config=/usr/jails/etc/rsyncd.conf
/usr/local/etc/rc.d/cbsdrsyncd: required_files:  -> 
Starting cbsdrsyncd.
Do you want to enable RACCT feature for resource accounting?
[yes(1) or no(0)]

[yes(1) or no(0)]
1
Shall i modify the /etc/rc.conf to sets cbsdd_enable=YES ?
[yes(1) or no(0)]

[yes(1) or no(0)]
1
/etc/rc.conf: cbsdd_enable:  -> YES
Shall i modify the /etc/rc.conf to sets rcshutdown_timeout="900"?
[yes(1) or no(0)]
1
/etc/rc.conf: rcshutdown_timeout: 90 -> 900
Shall i modify the /etc/sysctl.conf to sets kern.init_shutdown_timeout="900"?
[yes(1) or no(0)]
1
kern.init_shutdown_timeout: 120 -> 900
[Stage X: upgrading]
  * Prune legacy CBSD /usr/jails/etc/pfnat.conf config
>>> Done
  Congratulations! First CBSD initialization complete!

  Now your can run:
  service cbsdd start
  to run CBSD services.

  For change initenv settings in next time, use:
  cbsd initenv-tui

  Also don't forget to execute:
  cbsd initenv
  every time when you upgrade CBSD version.

  For an easy start:
  cbsd help

  General information:
  cbsd summary

  To start with jail:
  cbsd jcreate --help
  or: cbsd jconstruct-tui

  To start with bhyve:
  cbsd bcreate --help
  or: cbsd bconstruct-tui

  To start with XEN:
  cbsd xcreate --help
  or: cbsd xconstruct-tui

  To start with QEMU/NVMM:
  cbsd qcreate --help
  or: cbsd qconstruct-tui

  Enjoy CBSD!

preseedinit: Would you like a config for "cbsd init" preseed to be printed?
[yes(1) or no(0)]
1

---cut here ---
# cbsd initenv preseed file for fbhost host
# refer to the /usr/local/cbsd/share/initenv.conf
# for description.
#
nodeip="192.168.1.5"
jnameserver="114.114.114.114"
nodeippool="10.0.0.0/16"
nat_enable="pf"
fbsdrepo="1"
zfsfeat="1"
parallel="5"
stable="0"
sqlreplica="1"
statsd_bhyve_enable="0"
statsd_jail_enable="0"
statsd_hoster_enable="0"
ipfw_enable="1"
nodename="fbhost"
racct="1"
natip="192.168.1.5"
initenv_modify_sudoers="0"
initenv_modify_rcconf_hostname=""
initenv_modify_rcconf_cbsd_workdir="1"
initenv_modify_rcconf_cbsd_enable="1"
initenv_modify_rcconf_rcshutdown_timeout="1"
initenv_modify_syctl_rcshutdown_timeout="1"
initenv_modify_rcconf_cbsdrsyncd_enable="1"
initenv_modify_rcconf_cbsdrsyncd_flags="1"
initenv_modify_cbsd_homedir="1"
workdir="/usr/jails"
---end of cut---

创建一个jail

执行命令：

cbsd jconstruct-tui

首先进入配置画面

可以选择安装的软件包，cpu架构等。如果安装了qemu-devel,那么可以选择armv6、aarch64甚至riscv64       

配置好之后提示：

no base dir in: /usr/jails/basejail/base_amd64_amd64_14.1
Select base sources:
 0 .. CANCEL
 a .. build 
 b .. extract 
 c .. pkg 
 d .. populate 
 e .. repo 
选e         

自动去下载base.txz文件了

retrieve base.txz from download.freebsd.org, size: 198m
/usr/jails/tmp/src.71323/base.txz                      198 MB 2646 kBps 01m17s
然后到这一步有点慢，需要等一下：

Please wait: this will take a while...
Applying skel dir template from: /usr/jails/share/FreeBSD-jail-skel

最后安装完成：

To edit VM properties use: cbsd jconfig jname=jail1
To start VM use: cbsd jstart jail1
To stop VM use: cbsd jstop jail1
To remove VM use: cbsd jremove jail1
For attach VM console use: cbsd jlogin jail1

Creating jail1 complete: Enjoy!
/usr/jails/jails/jail1/etc already mounted
/usr/jails/jails/jail1/root already mounted
/usr/jails/jails/jail1/tmp already mounted
/usr/jails/jails/jail1/home already mounted
/usr/jails/jails/jail1/usr/local already mounted
/usr/jails/jails/jail1/compat already mounted
/usr/jails/jails/jail1/var already mounted
jcreate done in 6 minutes and 34 seconds
 

完成之后还有命令提醒，告知几个最常用的管理命令，真贴心！

启动CBSD jail

cbsd jstart jail1

Default NIC automatically selected: web
set resource limit: [ ]
jail renice: 1
Starting jail: jail1, parallel timeout=5
jail1: created
ELF ldconfig path: /lib /usr/lib /usr/lib/compat /usr/local/lib /usr/local/lib/compat/pkg /usr/local/lib/compat/pkg
32-bit compatibility ldconfig path: /usr/lib32 /usr/lib32
Updating /var/run/os-release done.
Creating and/or trimming log files.
Clearing /tmp (X related).
Updating motd:.
Starting syslogd.
Starting cron.

Fri Jun  7 00:13:02 CST 2024
CBSD setup: jail ipfw counters num: 99/100
jstart done in 5 seconds
启动也很快

登录 jail

cbsd jlogin jail1

速度好快

登进去之后，尽管无法ping通，但是网络是通了，可以使用pkg、curl 、wget等指令。太棒了！

管理 jail

使用cbsd --help获得命令列表

使用命令cbsd jconfig 来管理jail：

cbsd jconfig jname=jail1

使用cbsd jls 列出当前的jail

cbsd jls
JNAME  JID  IP4_ADDR  HOST_HOSTNAME    PATH                    STATUS
jail1  13   10.0.0.2  jail1.my.domain  /usr/jails/jails/jail1  On       

使用cbsd jstop 关闭jail

cbsd jstop  jail1
Stoping jail: jail1, parallel timeout=5
jstop done in 2 seconds

使用cbsd jstart 打开jail

cbsd jstart  jail1
Default NIC automatically selected: web
set resource limit: [ ]
jail renice: 1
Starting jail: jail1, parallel timeout=5
jail1: created
ELF ldconfig path: /lib /usr/lib /usr/lib/compat /usr/local/lib /usr/local/lib/compat/pkg /usr/local/lib/compat/pkg
32-bit compatibility ldconfig path: /usr/lib32 /usr/lib32
Updating /var/run/os-release done.
Creating and/or trimming log files.
Clearing /tmp (X related).
Updating motd:.
Starting syslogd.
Starting cron.

Fri Jun 14 12:43:21 CST 2024
CBSD setup: jail ipfw counters num: 99/100

使用cbsd jdestroy 删除jail

cbsd jdestroy jail1
jsnapshot: destroyed 0 snapshots in 0 seconds
jsnapshot: destroyed 0 snapshots in 0 seconds
jdestroy done in 2 seconds

使用cbsd destroy  删除jail 

另外直接执行cbsd指令，可以进入cbsd指令状态， 

CBSD的特点

同时支持jail和bhyve以及qemu，这使得它的应用范围非常广，比如创建riscv64开发环境。

CBSD速度很快，前期使用过AppJail和vm-bhyve等管理软件，CBSD是其中运行速度（响应速度）最快的。

CBSD官网提供的jail管理器生存周期统计

可以看到只有大约5种管理器还在持续更新，其它大部分都消亡了。CBDN是现存里面开发最早的。CBSD与其他项目的主要区别在于，CBSD被定位为完整的解决方案，具有尽可能简单的管理界面。

有很多人使用CBSD进行了很多有趣的工作，比如在FreeBSD上安装k8s：CBSD — FreeBSD Jail and Bhyve Management Tools

使用My-Bee管理k8s：GitHub - myb-project/guide: MyB Handbook 

基于CBSD的虚拟机管理平台ClonOS：Free Open-Source Hosting Platform «ClonOS»

使用bsdconfig进行CBSD config配置

使用“bsdconfig”文件进入FreeBSD管理界面后，最后一项竟然是“CBSD config” ，可以在这里面配置CBSD，真的没想到！ 

总结

这样使用下来，还是CBSD创建jail最简单方便啊，关键是创建了网络就通了，pkg还能用。

当然网络那块也可能是因为AppJail前期已经配好了网络导致的。大家可以尝试下，如果确实空白机不用网络配置就能通，请大家在下面留言说明。

CBSD是很罕见的下面只有一条调试记录的一个软件，对新手友好，确实可以拿来就用！

调试

初始化报错the CBSD requires a separate dataset different from the root one

Error: on ZFS-based systems, the CBSD requires a separate dataset different from the root one (zroot/ROOT/default).
Please create a separate dataset first, e.g.:

 /sbin/zfs create -o mountpoint=/usr/jails -o atime=off zroot/jails

Then re-run: env workdir=/usr/jails /usr/local/cbsd/sudoexec/initenv

原来是zfs系统需要先创建一个/usr/jails文件系统，按照提示使用下面语句创建：

 /sbin/zfs create -o mountpoint=/usr/jails -o atime=off zroot/jails