Linux system file machine-id missing

Background:

An issue happened on AIA REHL7 server when system is booting:

系统不能正常启动，输入root密码后进入维护模式，系统提示如下错误：

 systemd-journald[308]: Failed to open runtime journal: No such file or directory

 

Why server couldn't start normally?

System file /etc/machine-id is missing

 

How to solve this issue?

Copy /var/lib/dbus/machine-id to /etc/

 

How to prevent this issue?

1. Use system service to monitoring this file:

egenerate-machineid.service

[Unit]
Description=Generate New Machine ID
Documentation=man:systemd-firstboot(1)
DefaultDependencies=no
Conflicts=shutdown.target
After=systemd-remount-fs.service
Before=systemd-sysusers.service sysinit.target shutdown.target
ConditionPathIsReadWrite=/etc
ConditionPathExists=!/etc/machine-id

[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/usr/bin/systemd-firstboot --setup-machine-id

[Install]
WantedBy=sysinit.target

 

Creating service file under /etc/systemd/system/multi-user.target.wants/

[root@rh7 multi-user.target.wants]# ll
total 4
lrwxrwxrwx. 1 root root  35 Mar 29 17:49 atd.service -> /usr/lib/systemd/system/atd.service
lrwxrwxrwx. 1 root root  38 Jan 31  2019 auditd.service -> /usr/lib/systemd/system/auditd.service
lrwxrwxrwx. 1 root root  37 Jan 31  2019 crond.service -> /usr/lib/systemd/system/crond.service
-rwxr-x---. 1 root root 429 Sep  4 11:26 egenerate-machineid.service
lrwxrwxrwx. 1 root root  36 Jan 31  2019 etcd.service -> /usr/lib/systemd/system/etcd.service
lrwxrwxrwx. 1 root root  42 Jan 31  2019 irqbalance.service -> /usr/lib/systemd/system/irqbalance.service
lrwxrwxrwx. 1 root root  37 Jan 31  2019 kdump.service -> /usr/lib/systemd/system/kdump.service
lrwxrwxrwx. 1 root root  46 Jan 31  2019 NetworkManager.service -> /usr/lib/systemd/system/NetworkManager.service
lrwxrwxrwx. 1 root root  58 May 21 04:00 NetworkManager-wait-online.service -> /usr/lib/systemd/system/NetworkManager-wait-online.service
lrwxrwxrwx. 1 root root  39 Jan 31  2019 postfix.service -> /usr/lib/systemd/system/postfix.service
lrwxrwxrwx. 1 root root  40 Jan 31  2019 remote-fs.target -> /usr/lib/systemd/system/remote-fs.target
lrwxrwxrwx. 1 root root  41 Jan 31  2019 rhsmcertd.service -> /usr/lib/systemd/system/rhsmcertd.service
lrwxrwxrwx. 1 root root  39 Jan 31  2019 rsyslog.service -> /usr/lib/systemd/system/rsyslog.service
lrwxrwxrwx. 1 root root  36 Jan 31  2019 sshd.service -> /usr/lib/systemd/system/sshd.service
lrwxrwxrwx. 1 root root  39 Apr  7 13:56 sysstat.service -> /usr/lib/systemd/system/sysstat.service
lrwxrwxrwx. 1 root root  37 Jan 31  2019 tuned.service -> /usr/lib/systemd/system/tuned.service
lrwxrwxrwx. 1 root root  38 Mar 30 11:08 xinetd.service -> /usr/lib/systemd/system/xinetd.service
[root@rh7 multi-user.target.wants]# systemctl enable egenerate-machineid.service
Created symlink from /etc/systemd/system/sysinit.target.wants/egenerate-machineid.service to /etc/systemd/system/egenerate-machineid.service.
[root@rh7 multi-user.target.wants]# systemctl start egenerate-machineid.service
[root@rh7 multi-user.target.wants]# systemctl status egenerate-machineid.service
● egenerate-machineid.service - Generate New Machine ID
   Loaded: loaded (/etc/systemd/system/egenerate-machineid.service; enabled; vendor preset: disabled)
   Active: active (exited) since Wed 2019-09-04 11:26:52 CST; 6s ago
     Docs: man:systemd-firstboot(1)
  Process: 6271 ExecStart=/usr/bin/systemd-firstboot --setup-machine-id (code=exited, status=0/SUCCESS)
 Main PID: 6271 (code=exited, status=0/SUCCESS)

2. Check /var/log/boot.log after each booting, if there has any error messaging regarding journal Service eg: [DEPEND] Dependency failed for Flush Journal to Persistent Storage.

   you need to check if /etc/machine-id exist.

3. Check if /var/log/messages records any log after booting, if no new logs, need to check if /etc/machine-id exist.

4. Check systemd-journald status.