debian系统应用进程启动失败问题分析

现象描述

设备网络灯显示异常，通过串口登录到设备，启动网络服务时，系统显示信息如下：

root@linux:~# systemctl start network_5g
[  OK  ] Stopped Emergency Shell.
[FAILED] Failed to mount /proc/bus/usb.
See 'systemctl status proc-bus-usb.mount' for details.
[DEPEND] Dependency failed for Local File Systems.
[  OK  ] Started Emergency Shell.
[  OK  ] Reached target Emergency Mode.
[  OK  ] Stopped Dispatch Password ts to Console Directory Watch.
You are in emergency mode. After logging in, type "journalctl -xb" to view
system logs, "systemctl reboot" to reboot, "systemctl default" or "exit"
to boot into default mode.
Give root password for maintenance
(or press Control-D to continue):
Login incorrect

Give root password for maintenance
(or press Control-D to continue): 

输入root密码进入系统，执行journalctl -xb命令查看系统日志：

root@linux :/etc/systemd/system# journalctl -xe
Apr 06 10:14:59 linux mount[7779]: mount: /proc/bus/usb: mount point does not ex
Apr 06 10:14:59 linux systemd[1]: proc-bus-usb.mount: Failed with result 'exit-c
-- Subject: Unit failed
-- Defined-By: systemd
-- Support: https://www.debian.org/support
-- 
-- The unit proc-bus-usb.mount has entered the 'failed' state with result 'exit-
Apr 06 10:14:59 linux systemd[1]: Failed to mount /proc/bus/usb.
-- Subject: A start job for unit proc-bus-usb.mount has failed
-- Defined-By: systemd
-- Support: https://www.debian.org/support
-- 
-- A start job for unit proc-bus-usb.mount has finished with a failure.
-- 
-- The job identifier is 1055 and the job result is failed.
Apr 06 10:14:59 linux systemd[1]: Dependency failed for Local File Systems.
-- Subject: A start job for unit local-fs.target has failed
-- Defined-By: systemd
-- Support: https://www.debian.org/support
-- 
-- A start job for unit local-fs.target has finished with a failure.
-- 
-- The job identifier is 1052 and the job result is dependency.
lines 1110-1132/1132 (END)

分析

从日志中可以看出，直接原因是挂载 /proc/bus/usb失败，进而导致内核文件系统失败，

在systemd-remount-fs中是负责对/etc/fstab文件中解析的：

root@linux:/etc/systemd/system# systemctl status sresytemstemd-remoni=ount-fs.
 systemd-remount-fs.service - Remount Root and Kernel File Systems
   Loaded: loaded (/lib/systemd/system/systemd-remount-fs.service; static; vendo
   Active: active (exited) since Fri 2023-03-17 03:14:52 UTC; 2 weeks 6 days ago
     Docs: man:systemd-remount-fs.service(8)
           https://www.freedesktop.org/wiki/Software/systemd/APIFileSystems
 Main PID: 141 (code=exited, status=0/SUCCESS)
   CGroup: /system.slice/systemd-remount-fs.service

Warning: Journal has been rotated since unit was started. Log output is incomple
lines 1-9/9 (END)

 root@jenet:/etc/systemd/system# journalctl -xe
Apr 06 10:14:59 linux mount[7779]: mount: /proc/bus/usb: mount point does not ex
Apr 06 10:14:59 linux systemd[1]: proc-bus-usb.mount: Failed with result 'exit-c
-- Subject: Unit failed
-- Defined-By: systemd
-- Support: https://www.debian.org/support
-- 
-- The unit proc-bus-usb.mount has entered the 'failed' state with result 'exit-
Apr 06 10:14:59 linux systemd[1]: Failed to mount /proc/bus/usb.
-- Subject: A start job for unit proc-bus-usb.mount has failed
-- Defined-By: systemd
-- Support: https://www.debian.org/support
-- 
-- A start job for unit proc-bus-usb.mount has finished with a failure.
-- 
-- The job identifier is 1055 and the job result is failed.
Apr 06 10:14:59 linux systemd[1]: Dependency failed for Local File Systems.
-- Subject: A start job for unit local-fs.target has failed
-- Defined-By: systemd
-- Support: https://www.debian.org/support
-- 
-- A start job for unit local-fs.target has finished with a failure.
-- 
-- The job identifier is 1052 and the job result is dependency.
lines 1110-1132/1132 (END)

确实是 systemd-remount-fs服务启动失败导致的，查看/etc/fstab文件内容如下：

# UNCONFIGURED FSTAB FOR BASE SYSTEM
tmpfs /tmp tmpfs rw,nodev,size=1024m 0 0
/dev/mmcblk0p7 /linkedge ext4 defaults 0 0
/dev/mmcblk0p8 /config/worka ext4 defaults 0 0
/dev/mmcblk0p9 /config/workb ext4 defaults 0 0
/dev/mmcblk0p10 /config/work ext4 defaults 0 0
/dev/mmcblk0p11 /jetron/da ext4 defaults 0 0
/dev/mmcblk0p12 /config/log ext4 defaults 0 0
/dev/mmcblk0p13 /app ext4 defaults 0 0
none /proc/bus/usb usbfs defaults

出现了一条异常记录，将该条记录删除后，系统应用能正常启动，目前还不确定是谁向该文件中写入了此条记录。