rbd mirror服务无法实现开机自启动

现象

一个ceph集群机器重启后rbd-mirror服务没有自动重启。

[root@node-1-clz-1 ~]# systemctl enable ceph-rbd-mirror@admin
Created symlink from /etc/systemd/system/ceph-rbd-mirror.target.wants/ceph-rbd-mirror@admin.service to /usr/lib/systemd/system/ceph-rbd-mirror@.service.

systemctl enable ceph-rbd-mirror@admin 在/etc/systemd/system/ceph-rbd-mirror.target.wants目录下建立了软链接。

[root@node-1-clz-1 ~]# ls -l /etc/systemd/system/ceph-rbd-mirror.target.wants/ceph-rbd-mirror@admin.service
lrwxrwxrwx 1 root root 48 Jun 14 15:50 /etc/systemd/system/ceph-rbd-mirror.target.wants/ceph-rbd-mirror@admin.service -> /usr/lib/systemd/system/ceph-rbd-mirror@.service

重启后rbd mirror服务还是不能自动起来。

[root@node-1-clz-1 ~]# systemctl status ceph-rbd-mirror@admin
● ceph-rbd-mirror@admin.service - Ceph rbd mirror daemon
   Loaded: loaded (/usr/lib/systemd/system/ceph-rbd-mirror@.service; enabled; vendor preset: disabled)
   Active: inactive (dead)

对比osd mon,在/etc/systemd/system/ceph.target.wants/和/etc/systemd/system/multi-user.target.wants/目录也有软链接。

[root@node-1-clz-1 ~]# ls -l /etc/systemd/system/ceph.target.wants/
total 0
lrwxrwxrwx 1 root root 39 Apr 17 13:24 ceph-mds.target -> /usr/lib/systemd/system/ceph-mds.target
lrwxrwxrwx 1 root root 39 Apr 17 13:24 ceph-mgr.target -> /usr/lib/systemd/system/ceph-mgr.target
lrwxrwxrwx 1 root root 39 Apr 17 13:24 ceph-mon.target -> /usr/lib/systemd/system/ceph-mon.target
lrwxrwxrwx 1 root root 39 Apr 17 13:24 ceph-osd.target -> /usr/lib/systemd/system/ceph-osd.target

[root@node-1-clz-1 ~]# ls -l /etc/systemd/system/multi-user.target.wants/
total 0
lrwxrwxrwx. 1 root root 38 Apr 28  2017 auditd.service -> /usr/lib/systemd/system/auditd.service
lrwxrwxrwx  1 root root 39 Apr 17 13:24 ceph-mds.target -> /usr/lib/systemd/system/ceph-mds.target
lrwxrwxrwx  1 root root 39 Apr 17 13:24 ceph-mgr.target -> /usr/lib/systemd/system/ceph-mgr.target
lrwxrwxrwx  1 root root 39 Apr 17 13:24 ceph-mon.target -> /usr/lib/systemd/system/ceph-mon.target
lrwxrwxrwx  1 root root 39 Apr 17 13:24 ceph-osd.target -> /usr/lib/systemd/system/ceph-osd.target
lrwxrwxrwx  1 root root 35 Apr 17 13:24 ceph.target -> /usr/lib/systemd/system/ceph.target
lrwxrwxrwx. 1 root root 44 Apr 28  2017 cloud-config.service -> /usr/lib/systemd/system/cloud-config.service
lrwxrwxrwx. 1 root root 43 Apr 28  2017 cloud-final.service -> /usr/lib/systemd/system/cloud-final.service
lrwxrwxrwx. 1 root root 48 Apr 28  2017 cloud-init-local.service -> /usr/lib/systemd/system/cloud-init-local.service
lrwxrwxrwx. 1 root root 42 Apr 28  2017 cloud-init.service -> /usr/lib/systemd/system/cloud-init.service
lrwxrwxrwx. 1 root root 37 Apr 28  2017 crond.service -> /usr/lib/systemd/system/crond.service
lrwxrwxrwx. 1 root root 42 Apr 28  2017 irqbalance.service -> /usr/lib/systemd/system/irqbalance.service
lrwxrwxrwx. 1 root root 37 Apr 28  2017 kdump.service -> /usr/lib/systemd/system/kdump.service
lrwxrwxrwx. 1 root root 46 Apr 28  2017 NetworkManager.service -> /usr/lib/systemd/system/NetworkManager.service
lrwxrwxrwx. 1 root root 39 Apr 28  2017 postfix.service -> /usr/lib/systemd/system/postfix.service
lrwxrwxrwx. 1 root root 40 Apr 28  2017 remote-fs.target -> /usr/lib/systemd/system/remote-fs.target
lrwxrwxrwx. 1 root root 39 Apr 28  2017 rsyslog.service -> /usr/lib/systemd/system/rsyslog.service
lrwxrwxrwx. 1 root root 36 Apr 28  2017 sshd.service -> /usr/lib/systemd/system/sshd.service
lrwxrwxrwx. 1 root root 37 Apr 28  2017 tuned.service -> /usr/lib/systemd/system/tuned.service

使用systemctl enable ceph-rbd-mirror.target命令在/etc/systemd/system/multi-user.target.wants/目录下添加链接

[root@node-1-clz-1 ~]# ls -l /etc/systemd/system/ceph.target.wants/
total 0
lrwxrwxrwx 1 root root 39 Apr 17 13:24 ceph-mds.target -> /usr/lib/systemd/system/ceph-mds.target
lrwxrwxrwx 1 root root 39 Apr 17 13:24 ceph-mgr.target -> /usr/lib/systemd/system/ceph-mgr.target
lrwxrwxrwx 1 root root 39 Apr 17 13:24 ceph-mon.target -> /usr/lib/systemd/system/ceph-mon.target
lrwxrwxrwx 1 root root 39 Apr 17 13:24 ceph-osd.target -> /usr/lib/systemd/system/ceph-osd.target
lrwxrwxrwx 1 root root 46 Jun 14 16:34 ceph-rbd-mirror.target -> /usr/lib/systemd/system/ceph-rbd-mirror.target

[root@node-1-clz-1 ~]# ls -l /etc/systemd/system/multi-user.target.wants/
total 0
lrwxrwxrwx. 1 root root 38 Apr 28  2017 auditd.service -> /usr/lib/systemd/system/auditd.service
lrwxrwxrwx  1 root root 39 Apr 17 13:24 ceph-mds.target -> /usr/lib/systemd/system/ceph-mds.target
lrwxrwxrwx  1 root root 39 Apr 17 13:24 ceph-mgr.target -> /usr/lib/systemd/system/ceph-mgr.target
lrwxrwxrwx  1 root root 39 Apr 17 13:24 ceph-mon.target -> /usr/lib/systemd/system/ceph-mon.target
lrwxrwxrwx  1 root root 39 Apr 17 13:24 ceph-osd.target -> /usr/lib/systemd/system/ceph-osd.target
lrwxrwxrwx  1 root root 46 Jun 14 16:34 ceph-rbd-mirror.target -> /usr/lib/systemd/system/ceph-rbd-mirror.target
lrwxrwxrwx  1 root root 35 Apr 17 13:24 ceph.target -> /usr/lib/systemd/system/ceph.target
lrwxrwxrwx. 1 root root 44 Apr 28  2017 cloud-config.service -> /usr/lib/systemd/system/cloud-config.service
lrwxrwxrwx. 1 root root 43 Apr 28  2017 cloud-final.service -> /usr/lib/systemd/system/cloud-final.service
lrwxrwxrwx. 1 root root 48 Apr 28  2017 cloud-init-local.service -> /usr/lib/systemd/system/cloud-init-local.service
lrwxrwxrwx. 1 root root 42 Apr 28  2017 cloud-init.service -> /usr/lib/systemd/system/cloud-init.service
lrwxrwxrwx. 1 root root 37 Apr 28  2017 crond.service -> /usr/lib/systemd/system/crond.service
lrwxrwxrwx. 1 root root 42 Apr 28  2017 irqbalance.service -> /usr/lib/systemd/system/irqbalance.service
lrwxrwxrwx. 1 root root 37 Apr 28  2017 kdump.service -> /usr/lib/systemd/system/kdump.service
lrwxrwxrwx. 1 root root 46 Apr 28  2017 NetworkManager.service -> /usr/lib/systemd/system/NetworkManager.service
lrwxrwxrwx. 1 root root 39 Apr 28  2017 postfix.service -> /usr/lib/systemd/system/postfix.service
lrwxrwxrwx. 1 root root 40 Apr 28  2017 remote-fs.target -> /usr/lib/systemd/system/remote-fs.target
lrwxrwxrwx. 1 root root 39 Apr 28  2017 rsyslog.service -> /usr/lib/systemd/system/rsyslog.service
lrwxrwxrwx. 1 root root 36 Apr 28  2017 sshd.service -> /usr/lib/systemd/system/sshd.service
lrwxrwxrwx. 1 root root 37 Apr 28  2017 tuned.service -> /usr/lib/systemd/system/tuned.service

重启节点,开机自启动。

[root@node-1-clz-1 ~]# systemctl status ceph-rbd-mirror@admin
● ceph-rbd-mirror@admin.service - Ceph rbd mirror daemon
   Loaded: loaded (/usr/lib/systemd/system/ceph-rbd-mirror@.service; enabled; vendor preset: disabled)
   Active: active (running) since Thu 2018-06-14 16:39:51 CST; 50s ago
 Main PID: 949 (rbd-mirror)
   CGroup: /system.slice/system-ceph\x2drbd\x2dmirror.slice/ceph-rbd-mirror@admin.service
           └─949 /usr/bin/rbd-mirror -f --cluster ceph --id admin --setuser ceph --setgroup ceph

Jun 14 16:39:51 node-1-clz-1.novalocal systemd[1]: Started Ceph rbd mirror daemon.
Jun 14 16:39:51 node-1-clz-1.novalocal systemd[1]: Starting Ceph rbd mirror daemon...

原因

因为在同一个节点上有多个ceph-osd的服务，为了能够统一启动和关闭ceph-osd服务，所以ceph中所有的服务统一格式为*.service和*.target，在*.service中配置partof成为*.target组的一个服务。WantedBy=multi-user.target都是在*.target中，这样可以一起设置osd开机启动。WantedBy=multi-user.target会在systemctl enable *.target是在/etc/systemd/system/multi-user.target.wants/目录下生成对应的软连接，在开机时自启。
所有的ceph服务包括ceph-rbd-mirror为了统一格式，都使用的该方法配置对应的systemd服务。ceph-mon,ceph-osd,ceph-mgr已经在spec中定义安装后执行systemctl enable ceph-mon.target,systemctl enable ceph-osd.target实现开机自启。rbd-mirror服务不是所有集群必须启动，所以需要手动systemctl enable ceph-rbd-mirror.target,设置开机自启。
###2、在muli-usr.target目录下添加软链接
WantedBy=multi-user.target ceph.target 会在enable的时候在/etc/systemd/system/multi-user.target.wants/目录下生成软链接，而ceph所有的服务wantedby都是写在了vim /usr/lib/systemd/system/ceph-rbd-mirror.target而不是/usr/lib/systemd/system/ceph-rbd-mirror@.service里。所以systemctl enable ceph-rbd-mirror@admin不能设置开机启动。

#vim /usr/lib/systemd/system/ceph-rbd-mirror.target
[Unit]
Description=ceph target allowing to start/stop all ceph-rbd-mirror@.service instances at once
PartOf=ceph.target
Before=ceph.target
[Install]
WantedBy=multi-user.target ceph.target

为什么不写在ceph-rbd-mirror@.service:
一般一个host只有一个服务，但是ceph不同。ceph-osd的服务在同一个host有多个，如果在*@.service加WantedBy=multi-user.target ceph.target，当需要设置开机启动时需要一个一个的osd服务去enable，如果放在*.target里就可以使用一条命令：systemctl enabled ceph-osd.target即可，其他服务也统一了标准。
.target如何在启动时启动所有的.service
其中PartOf=ceph-rbd-mirror.target表示时ceph-rbd-mirror.target组中的一个组员，在启动/关闭target时会启动/关闭它。这一点对于osd很方便。

[root@node-1 multi-user.target.wants]# cat /usr/lib/systemd/system/ceph-rbd-mirror@.service
[Unit]
Description=Ceph rbd mirror daemon
After=network-online.target local-fs.target
Wants=network-online.target local-fs.target
PartOf=ceph-rbd-mirror.target

[Service]
LimitNOFILE=1048576
LimitNPROC=1048576
EnvironmentFile=-/etc/sysconfig/ceph
Environment=CLUSTER=ceph
ExecStart=/usr/bin/rbd-mirror -f --cluster ${CLUSTER} --id %i --setuser ceph --setgroup ceph -c /etc/ceph/rbdmirror/remote.conf
ExecReload=/bin/kill -HUP $MAINPID
PrivateDevices=yes
ProtectHome=true
ProtectSystem=full
PrivateTmp=true
Restart=on-failure
StartLimitInterval=30min
StartLimitBurst=3
TasksMax=infinity

[Install]
WantedBy=multi-user.target