运行Docker容器的主机因磁盘没有剩余空间而宕机问题排查

1，使用

   df -h //查询整体磁盘使用情况
  
  /dev/vda1        20G   19G     0 100% /
  none            4.0K     0  4.0K   0% /sys/fs/cgroup
  udev            3.9G  4.0K  3.9G   1% /dev
  tmpfs           799M  1.6M  797M   1% /run
  none            5.0M     0  5.0M   0% /run/lock
  none            3.9G     0  3.9G   0% /run/shm
  none            100M     0  100M   0% /run/user
  /dev/vdb        197G  5.2G  182G   3% /data

发现/dev/vda1磁盘占满

2，依次查询哪些文件过大，一般就是服务写入的log 日志比较多

     例如：cd /usr
    使用 du -sh *  来查询

    344M    bin
    4.0K    games
    25M     include
    480M    lib
    28M     local
    13M     sbin
    286M    share
    2.6G    src



找到占用空间过大的日志文件夹 清除就ok了

3,再次查询

  df -h

 /dev/vda1        20G   12G  7.0G  63% /
  none            4.0K     0  4.0K   0% /sys/fs/cgroup
  udev            3.9G  4.0K  3.9G   1% /dev
  tmpfs           799M  1.6M  797M   1% /run
  none            5.0M     0  5.0M   0% /run/lock
  none            3.9G     0  3.9G   0% /run/shm
  none            100M     0  100M   0% /run/user
  /dev/vdb        197G  5.2G  182G   3% /data

 具体如何删除内核，这里就不做详述了。

另外为防止下次出现类似情况，设置docker日志机制

vi /etc/docker/daemon.json

{
"registry-mirrors": ["https://lo649qoz.mirror.aliyuncs.com"],
"log-driver":"json-file",
"log-opts": { "max-size": "100m", "max-file": "5" }
}

vi /etc/sysconfig/docker

# /etc/sysconfig/docker

# Modify these options if you want to change the way the docker daemon runs
OPTIONS='--selinux-enabled --log-driver=journald --signature-verification=false'
if [ -z "${DOCKER_CERT_PATH}" ]; then
    DOCKER_CERT_PATH=/etc/docker
fi

# Do not add registries in this file anymore. Use /etc/containers/registries.conf
# instead. For more information reference the registries.conf(5) man page.

# Location used for temporary files, such as those created by
# docker load and build operations. Default is /var/lib/docker/tmp
# Can be overriden by setting the following environment variable.
# DOCKER_TMPDIR=/var/tmp

# Controls the /etc/cron.daily/docker-logrotate cron job status.
# To disable, uncomment the line below.
# LOGROTATE=false

# docker-latest daemon can be used by starting the docker-latest unitfile.
# To use docker-latest client, uncomment below lines
#DOCKERBINARY=/usr/bin/docker-latest
#DOCKERDBINARY=/usr/bin/dockerd-latest
#DOCKER_CONTAINERD_BINARY=/usr/bin/docker-containerd-latest
#DOCKER_CONTAINERD_SHIM_BINARY=/usr/bin/docker-containerd-shim-latest

删除红色字体“--log-driver=journald” 然后保存

执行命令重启docker:

systemctl daemon-reload

systemctl restart docker