一次磁盘满导致gitlab无法访问 的bug

同事说，gitlab无法访问，访问发现502 ，去服务器上看了下nginx error.log

2018/01/31 15:01:50 [alert] 16867#0: *67473 write() to "/usr/local/nginx/logs/access.log" failed (28: No space left on device) while logging request, client: 180.169.120.218, server: gitlab.imacco.com, request: "GET /favicon.ico HTTP/1.1", upstream: "http://127.0.0.1:8080/favicon.ico", host: "gitlab.imacco.com", referrer: "http://gitlab.imacco.com/“


#关键部分
# write() to "/usr/local/nginx/logs/access.log" failed (28: No space left on device

提示，access.log写入失败，磁盘没有空间了。查看磁盘空间，寻找大文件，删除，解决。

# 查看磁盘容量和使用情况
df -lh

[root@iZbp178t84wkep4cvir0zsZ logs] $ df -lh
Filesystem      Size  Used Avail Use% Mounted on
/dev/vda1        40G   38G     0 100% /
devtmpfs        3.9G     0  3.9G   0% /dev
tmpfs           3.9G     0  3.9G   0% /dev/shm
tmpfs           3.9G   67M  3.8G   2% /run
tmpfs           3.9G     0  3.9G   0% /sys/fs/cgroup
tmpfs           783M     0  783M   0% /run/user/0
overlay          40G   38G     0 100% /var/lib/docker/overlay/f3eaa0fb4aa3596ad7debc5fed48ddd7edc8a9095bc4b48e70f856a73002a406/merged
shm              64M     0   64M   0% /var/lib/docker/containers/5f9726fc0da5097fe0e731e0675e9a814392b1bdf9ec4a978af219c199687feb/shm

# 寻找当前目录大文件
du -sh *

[root@iZbp178t84wkep4cvir0zsZ /] $ du -sh *
6.4G    1509434370_gitlab_backup.tar
0    bin
145M    boot
4.0K    deployment
0    dev
37M    etc
14G    gitlabnew
4.0K    home
0    lib
0    lib64
16K    lost+found
352M    mail
4.0K    media
4.0K    mnt
4.0K    opt
44K    postfix.sql
du: cannot access ‘proc/3171/task/7826’: No such file or directory
du: cannot access ‘proc/7746/task/7746/fd/4’: No such file or directory
du: cannot access ‘proc/7746/task/7746/fdinfo/4’: No such file or directory
du: cannot access ‘proc/7746/fd/4’: No such file or directory
du: cannot access ‘proc/7746/fdinfo/4’: No such file or directory
0    proc
1.4G    root
67M    run
0    sbin
4.0K    srv
0    sys
72K    tmp
1.6G    usr

# 删除了 6.4G的备份文件 。重新访问 恢复正常访问