现象
一个ceph集群机器重启后rbd-mirror服务没有自动重启。
[root@node-1-clz-1 ~]# systemctl enable ceph-rbd-mirror@admin
Created symlink from /etc/systemd/system/ceph-rbd-mirror.target.wants/ceph-rbd-mirror@admin.service to /usr/lib/systemd/system/ceph-rbd-mirror@.service.
systemctl enable ceph-rbd-mirror@admin 在/etc/systemd/system/ceph-rbd-mirror.target.wants目录下建立了软链接。
[root@node-1-clz-1 ~]# ls -l /etc/systemd/system/ceph-rbd-mirror.target.wants/ceph-rbd-mirror@admin.service
lrwxrwxrwx 1 root root 48 Jun 14 15:50 /etc/systemd/system/ceph-rbd-mirror.target.wants/ceph-rbd-mirror@admin.service -> /usr/lib/systemd/system/ceph-rbd-mirror@.service
重启后rbd mirror服务还是不能自动起来。
[root@node-1-clz-1 ~]# systemctl status ceph-rbd-mirror@admin
● ceph-rbd-mirror@admin.service - Ceph rbd mirror daemon
Loaded: loaded (/usr/lib/systemd/system/ceph-rbd-mirror@.service; enabled; vendor preset: disabled)
Active: inactive (dead)
对比osd mon,在/etc/systemd/system/ceph.target.wants/和/etc/systemd/system/multi-user.target.wants/目录也有软链接。
[root@node-1-clz-1 ~]# ls -l /etc/systemd/system/ceph.target.wants/
total 0
lrwxrwxrwx 1 root root 39 Apr 17 13:24 ceph-mds.target -> /usr/lib/systemd/system/ceph-mds.target
lrwxrwxrwx 1 root root 39 Apr 17 13:24 ceph-mgr.target -> /usr/lib/systemd/system/ceph-mgr.target
lrwxrwxrwx 1 root root 39 Apr 17 13:24 ceph-mon.target -> /usr/lib/systemd/system/ceph-mon.target
lrwxrwxrwx 1 root root 39 Apr 17 13:24 ceph-osd.target -> /usr/lib/systemd/system/ceph-osd.target
[root@node-1-clz-1 ~]# ls -l /etc/systemd/system/multi-user.target.wants/
total 0
lrwxrwxrwx. 1 root root 38 Apr 28 2017 auditd.service -> /usr/lib/systemd/system/auditd.service
lrwxrwxrwx 1 root root 39 Apr 17 13:24 ceph-mds.target -> /usr/lib/systemd/system/ceph-mds.target
lrwxrwxrwx 1 root root 39 Apr 17 13:24 ceph-mgr.target -> /usr/lib/systemd/system/ceph-mgr.target
lrwxrwxrwx 1 root root 39 Apr 17 13:24 ceph-mon.target -> /usr/lib/systemd/system/ceph-mon.target
lrwxrwxrwx 1 root root 39 Apr 17 13:24 ceph-osd.target -> /usr/lib/systemd/system/ceph-osd.target
lrwxrwxrwx 1 root root 35 Apr 17 13:24 ceph.target -> /usr/lib/systemd/system/ceph.target
lrwxrwxrwx. 1 root root 44 Apr 28 2017 cloud-config.service -> /usr/lib/systemd/system/cloud-config.service
lrwxrwxrwx. 1 root root 43 Apr 28 2017 cloud-final.service -> /usr/lib/systemd/system/cloud-final.service
lrwxrwxrwx. 1 root root 48 Apr 28 2017 cloud-init-local.service -> /usr/lib/systemd/system/cloud-init-local.service
lrwxrwxrwx. 1 root root 42 Apr 28 2017 cloud-init.service -> /usr/lib/systemd/system/cloud-init.service
lrwxrwxrwx. 1 root root 37 Apr 28 2017 crond.service -> /usr/lib/systemd/system/crond.service
lrwxrwxrwx. 1 root root 42 Apr 28 2017 irqbalance.service -> /usr/lib/systemd/system/irqbalance.service
lrwxrwxrwx. 1 root root 37 Apr 28 2017 kdump.service -> /usr/lib/systemd/system/kdump.service
lrwxrwxrwx. 1 root root 46 Apr 28 2017 NetworkManager.service -> /usr/lib/systemd/system/NetworkManager.service
lrwxrwxrwx. 1 root root 39 Apr 28 2017 postfix.service -> /usr/lib/systemd/system/postfix.service
lrwxrwxrwx. 1 root root 40 Apr 28 2017 remote-fs.target -> /usr/lib/systemd/system/remote-fs.target
lrwxrwxrwx. 1 root root 39 Apr 28 2017 rsyslog.service -> /usr/lib/systemd/system/rsyslog.service
lrwxrwxrwx. 1 root root 36 Apr 28 2017 sshd.service -> /usr/lib/systemd/system/sshd.service
lrwxrwxrwx. 1 root root 37 Apr 28 2017 tuned.service -> /usr/lib/systemd/system/tuned.service
使用systemctl enable ceph-rbd-mirror.target命令在/etc/systemd/system/multi-user.target.wants/目录下添加链接
[root@node-1-clz-1 ~]# ls -l /etc/systemd/system/ceph.target.wants/
total 0
lrwxrwxrwx 1 root root 39 Apr 17 13:24 ceph-mds.target -> /usr/lib/systemd/system/ceph-mds.target
lrwxrwxrwx 1 root root 39 Apr 17 13:24 ceph-mgr.target -> /usr/lib/systemd/system/ceph-mgr.target
lrwxrwxrwx 1 root root 39 Apr 17 13:24 ceph-mon.target -> /usr/lib/systemd/system/ceph-mon.target
lrwxrwxrwx 1 root root 39 Apr 17 13:24 ceph-osd.target -> /usr/lib/systemd/system/ceph-osd.target
lrwxrwxrwx 1 root root 46 Jun 14 16:34 ceph-rbd-mirror.target -> /usr/lib/systemd/system/ceph-rbd-mirror.target
[root@node-1-clz-1 ~]# ls -l /etc/systemd/system/multi-user.target.wants/
total 0
lrwxrwxrwx. 1 root root 38 Apr 28 2017 auditd.service -> /usr/lib/systemd/system/auditd.service
lrwxrwxrwx 1 root root 39 Apr 17 13:24 ceph-mds.target -> /usr/lib/systemd/system/ceph-mds.target
lrwxrwxrwx 1 root root 39 Apr 17 13:24 ceph-mgr.target -> /usr/lib/systemd/system/ceph-mgr.target
lrwxrwxrwx 1 root root 39 Apr 17 13:24 ceph-mon.target -> /usr/lib/systemd/system/ceph-mon.target
lrwxrwxrwx 1 root root 39 Apr 17 13:24 ceph-osd.target -> /usr/lib/systemd/system/ceph-osd.target
lrwxrwxrwx 1 root root 46 Jun 14 16:34 ceph-rbd-mirror.target -> /usr/lib/systemd/system/ceph-rbd-mirror.target
lrwxrwxrwx 1 root root 35 Apr 17 13:24 ceph.target -> /usr/lib/systemd/system/ceph.target
lrwxrwxrwx. 1 root root 44 Apr 28 2017 cloud-config.service -> /usr/lib/systemd/system/cloud-config.service
lrwxrwxrwx. 1 root root 43 Apr 28 2017 cloud-final.service -> /usr/lib/systemd/system/cloud-final.service
lrwxrwxrwx. 1 root root 48 Apr 28 2017 cloud-init-local.service -> /usr/lib/systemd/system/cloud-init-local.service
lrwxrwxrwx. 1 root root 42 Apr 28 2017 cloud-init.service -> /usr/lib/systemd/system/cloud-init.service
lrwxrwxrwx. 1 root root 37 Apr 28 2017 crond.service -> /usr/lib/systemd/system/crond.service
lrwxrwxrwx. 1 root root 42 Apr 28 2017 irqbalance.service -> /usr/lib/systemd/system/irqbalance.service
lrwxrwxrwx. 1 root root 37 Apr 28 2017 kdump.service -> /usr/lib/systemd/system/kdump.service
lrwxrwxrwx. 1 root root 46 Apr 28 2017 NetworkManager.service -> /usr/lib/systemd/system/NetworkManager.service
lrwxrwxrwx. 1 root root 39 Apr 28 2017 postfix.service -> /usr/lib/systemd/system/postfix.service
lrwxrwxrwx. 1 root root 40 Apr 28 2017 remote-fs.target -> /usr/lib/systemd/system/remote-fs.target
lrwxrwxrwx. 1 root root 39 Apr 28 2017 rsyslog.service -> /usr/lib/systemd/system/rsyslog.service
lrwxrwxrwx. 1 root root 36 Apr 28 2017 sshd.service -> /usr/lib/systemd/system/sshd.service
lrwxrwxrwx. 1 root root 37 Apr 28 2017 tuned.service -> /usr/lib/systemd/system/tuned.service
重启节点,开机自启动。
[root@node-1-clz-1 ~]# systemctl status ceph-rbd-mirror@admin
● ceph-rbd-mirror@admin.service - Ceph rbd mirror daemon
Loaded: loaded (/usr/lib/systemd/system/ceph-rbd-mirror@.service; enabled; vendor preset: disabled)
Active: active (running) since Thu 2018-06-14 16:39:51 CST; 50s ago
Main PID: 949 (rbd-mirror)
CGroup: /system.slice/system-ceph\x2drbd\x2dmirror.slice/ceph-rbd-mirror@admin.service
└─949 /usr/bin/rbd-mirror -f --cluster ceph --id admin --setuser ceph --setgroup ceph
Jun 14 16:39:51 node-1-clz-1.novalocal systemd[1]: Started Ceph rbd mirror daemon.
Jun 14 16:39:51 node-1-clz-1.novalocal systemd[1]: Starting Ceph rbd mirror daemon...
原因
因为在同一个节点上有多个ceph-osd的服务,为了能够统一启动和关闭ceph-osd服务,所以ceph中所有的服务统一格式为*.service和*.target,在*.service中配置partof成为*.target组的一个服务。WantedBy=multi-user.target都是在*.target中,这样可以一起设置osd开机启动。WantedBy=multi-user.target会在systemctl enable *.target是在/etc/systemd/system/multi-user.target.wants/目录下生成对应的软连接,在开机时自启。
所有的ceph服务包括ceph-rbd-mirror为了统一格式,都使用的该方法配置对应的systemd服务。ceph-mon,ceph-osd,ceph-mgr已经在spec中定义安装后执行systemctl enable ceph-mon.target,systemctl enable ceph-osd.target实现开机自启。rbd-mirror服务不是所有集群必须启动,所以需要手动systemctl enable ceph-rbd-mirror.target,设置开机自启。
###2、在muli-usr.target目录下添加软链接
WantedBy=multi-user.target ceph.target 会在enable的时候在/etc/systemd/system/multi-user.target.wants/目录下生成软链接,而ceph所有的服务wantedby都是写在了vim /usr/lib/systemd/system/ceph-rbd-mirror.target而不是/usr/lib/systemd/system/ceph-rbd-mirror@.service里。所以systemctl enable ceph-rbd-mirror@admin不能设置开机启动。
#vim /usr/lib/systemd/system/ceph-rbd-mirror.target
[Unit]
Description=ceph target allowing to start/stop all ceph-rbd-mirror@.service instances at once
PartOf=ceph.target
Before=ceph.target
[Install]
WantedBy=multi-user.target ceph.target
为什么不写在ceph-rbd-mirror@.service:
一般一个host只有一个服务,但是ceph不同。ceph-osd的服务在同一个host有多个,如果在*@.service加WantedBy=multi-user.target ceph.target,当需要设置开机启动时需要一个一个的osd服务去enable,如果放在*.target里就可以使用一条命令:systemctl enabled ceph-osd.target即可,其他服务也统一了标准。
.target如何在启动时启动所有的.service
其中PartOf=ceph-rbd-mirror.target表示时ceph-rbd-mirror.target组中的一个组员,在启动/关闭target时会启动/关闭它。这一点对于osd很方便。
[root@node-1 multi-user.target.wants]# cat /usr/lib/systemd/system/ceph-rbd-mirror@.service
[Unit]
Description=Ceph rbd mirror daemon
After=network-online.target local-fs.target
Wants=network-online.target local-fs.target
PartOf=ceph-rbd-mirror.target
[Service]
LimitNOFILE=1048576
LimitNPROC=1048576
EnvironmentFile=-/etc/sysconfig/ceph
Environment=CLUSTER=ceph
ExecStart=/usr/bin/rbd-mirror -f --cluster ${CLUSTER} --id %i --setuser ceph --setgroup ceph -c /etc/ceph/rbdmirror/remote.conf
ExecReload=/bin/kill -HUP $MAINPID
PrivateDevices=yes
ProtectHome=true
ProtectSystem=full
PrivateTmp=true
Restart=on-failure
StartLimitInterval=30min
StartLimitBurst=3
TasksMax=infinity
[Install]
WantedBy=multi-user.target