Ubuntu下MongoDB failed code=exited, status=XXX问题排查 以及 Ubuntu下查看磁盘使用情况和快速清理方法

问题描述：早上泡了杯茶，前端同学通知我服务不可用了。艾，昨天还能用的，今天怎么凉了，应该不是代码出了什么问题，感觉问题出现在外部。上服务器查看了下今天的日志，今天的 error log文件有400K，往常都是个位数的，查看之，果然，MongoDB无法连接，频繁报错。

问题排查：服务器用的是 Ubuntu，通过  service mongod status 查看mongod的状态如下：

 mongod.service - MongoDB Database Server
   Loaded: loaded (/lib/systemd/system/mongod.service; enabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since Tue 2018-08-28 22:17:37 CEST; 10s ago
     Docs: https://docs.mongodb.org/manual
  Process: 392 ExecStart=/usr/bin/mongod --config /etc/mongod.conf (code=exited, status=48)
 Main PID: 392 (code=exited, status=48)
      CPU: 33ms

aug 28 22:17:37 staging-manager-1 systemd[1]: Started MongoDB Database Server.
aug 28 22:17:37 staging-manager-1 mongod[392]: 2018-08-28T22:17:37.389+0200 I CONTROL  [main] Automatically disabling TLS 1.0, to fo
aug 28 22:17:37 staging-manager-1 systemd[1]: mongod.service: Main process exited, code=exited, status=48/n/a
aug 28 22:17:37 staging-manager-1 systemd[1]: mongod.service: Unit entered failed state.
aug 28 22:17:37 staging-manager-1 systemd[1]: mongod.service: Failed with result 'exit-code'.

从上状态可以看到 MongoDB的启动命令如下

/usr/bin/mongod --config /etc/mongod.conf

接下来去查看 mongod.conf 配置文件，从中可以看到 MongoDB日志的位置

# where to write logging data.
systemLog:
  destination: file
  logAppend: true
  path: /var/log/mongodb/mongod.log

找到了MongoDB系统日志的位置，去查看这个日志就可以看到具体的报错信息了，如下：

2019-02-19T09:18:11.786+0800 E STORAGE  [WTCheckpointThread] WiredTiger error (28) [1550539091:784271][1123:0x7fea309b1700], file:WiredTiger.wt, WT_SESSION.checkpoint: __posix_file_write, 579: /var/lib/mongodb/WiredTiger.turtle.set: handle-write: pwrite: failed to write 1077 bytes at offset 0: No space left on device Raw: [1550539091:784271][1123:0x7fea309b1700], file:WiredTiger.wt, WT_SESSION.checkpoint: __posix_file_write, 579: /var/lib/mongodb/WiredTiger.turtle.set: handle-write: pwrite: failed to write 1077 bytes at offset 0: No space left on device

直接看最后的： No space left on device。  哦，特喵的测试服磁盘又满了。

分析磁盘占用、找到大文件快速清理看这篇：https://blog.csdn.net/weixin_37882382/article/details/87704533

然后，终于清理完成了，有空间了，开始重启 MongoDB：

service mongod restart

艾，用 status命令查看又没有成功，再去看日志，报如下错误：

2019-02-19T10:48:35.735+0800 E STORAGE  [initandlisten] WiredTiger error (13) [1550544515:735833][8897:0x7f2977d70a40], wiredtiger_open: __posix_open_file, 715: /var/lib/mongodb/WiredTiger.turtle: handle-open: open: Permission denied Raw: [1550544515:735833][8897:0x7f2977d70a40], wiredtiger_open: __posix_open_file, 715: /var/lib/mongodb/WiredTiger.turtle: handle-open: open: Permission denied

意思为：mongod无权限访问 /var/lib/mongodb目录，好的  分配权限命令如下：

chown -R mongodb:mongodb /var/lib/mongodb

再重启，终于  active了，over