Centos磁盘占满

凡解

已于 2023-05-03 19:52:55 修改

阅读量1.4k

点赞数

分类专栏： # Docker容器化文章标签： centos docker

于 2022-07-25 16:01:14 首次发布

本文链接：https://blog.csdn.net/Abraxs/article/details/125974376

版权

Docker容器化专栏收录该内容

14 篇文章 0 订阅

订阅专栏

场景:

想拷贝一份表作为备份，出现错误语句；
Create table/tablespace ‘table_xx’ faild,as disk is full
disk	英[dɪsk] 磁盘; 磁碟;

排查操作

df -hl 查看磁盘使用情况

[root@cc containers]# df -hl
Filesystem      Size  Used Avail Use% Mounted on
devtmpfs        901M     0  901M   0% /dev
tmpfs           915M     0  915M   0% /dev/shm
tmpfs           915M   92M  823M  11% /run
tmpfs           915M     0  915M   0% /sys/fs/cgroup
/dev/vda1        40G   40G   20K 100% /
overlay          40G   40G   20K 100% /var/lib/docker/overlay2/a38e23ba6f22ac490ad863ebd65965dad47ad717abeda7d56691ea6d00a2a2ec/merged
tmpfs           183M     0  183M   0% /run/user/0

overlay          40G   40G   20K 100% /var/lib/docker/overlay2/a38e23ba6f22ac490ad863ebd65965dad47ad717abeda7d56691ea6d00a2a2ec/merged

可以看出是和docker容器相关所导致(实际就两镜像两容器，mysql 和 mongo ，mysql表比较多，开始还以为是mysql磁盘占用多了，最后发现是mongo)；

也可以通过命令(du -h -x --max-depth=1)一层层去排查(此命令是查看各文件夹所占空间大小)：
du相关命令还有很多，有根据条件搜索的；

步骤：
切换到根目录；

[root@cc/]# du -h -x --max-depth=1
24M	./etc
585M	./root
36G	./var
2.8G	./usr
117M	./boot
0	./home
0	./media
0	./mnt
0	./opt
0	./srv
0	./tmp
12K	./data
0	./nacos
40G	.
[root@cc/]# ccd /.
-bash: ccd: command not found
[root@cc/]# ls 
bin  boot  data  dev  etc  gitlab-ce-11.10.0-ce.0.el6.x86_64.rpm  home  lib  lib64  media  mnt  nacos  opt  proc  root  run  sbin  srv  sys  tmp  usr  var
[root@cc/]# cd var/
[root@cc var]# ls 
adm  cache  crash  ctcss  ctcss-tmp  db  empty  ftp  games  gopher  kerberos  lib  local  lock  log  mail  nis  opt  preserve  run  spool  tmp  yp
[root@cc var]# ll
total 8
drwxr-xr-x.  2 root root    6 May 11  2019 adm
drwxr-xr-x.  8 root root   91 Mar 29  2021 cache
drwxr-xr-x.  2 root root    6 May 11  2019 crash
drwxr-xr-x  14 root root  169 Mar 29  2021 ctcss
drwxr-xr-x   2 root root   42 Mar 29  2021 ctcss-tmp
drwxr-xr-x.  3 root root   18 Mar 29  2021 db
drwxr-xr-x.  3 root root   18 Mar  8  2021 empty
drwxr-xr-x.  2 root root    6 May 11  2019 ftp
drwxr-xr-x.  2 root root    6 May 11  2019 games
drwxr-xr-x.  2 root root    6 May 11  2019 gopher
drwxr-xr-x.  3 root root   18 Mar  8  2021 kerberos
drwxr-xr-x. 31 root root 4096 Oct  1  2021 lib
drwxr-xr-x.  2 root root    6 May 11  2019 local
lrwxrwxrwx.  1 root root   11 Mar  8  2021 lock -> ../run/lock
drwxr-xr-x. 10 root root 4096 Jul 17 03:21 log
lrwxrwxrwx.  1 root root   10 May 11  2019 mail -> spool/mail
drwxr-xr-x.  2 root root    6 May 11  2019 nis
drwxr-xr-x.  2 root root    6 May 11  2019 opt
drwxr-xr-x.  2 root root    6 May 11  2019 preserve
lrwxrwxrwx.  1 root root    6 Mar  8  2021 run -> ../run
drwxr-xr-x.  8 root root   87 Mar  8  2021 spool
drwxrwxrwt.  3 root root   85 Dec 17  2021 tmp
drwxr-xr-x.  2 root root    6 May 11  2019 yp
[root@cc var]# ls 
adm  cache  crash  ctcss  ctcss-tmp  db  empty  ftp  games  gopher  kerberos  lib  local  lock  log  mail  nis  opt  preserve  run  spool  tmp  yp
[root@cc var]# du -h --max-depth=1
36G	./lib
489M	./log
0	./adm
57M	./cache
0	./db
0	./empty
0	./ftp
0	./games
0	./gopher
0	./local
0	./nis
0	./opt
0	./preserve
12K	./spool
0	./tmp
0	./yp
0	./kerberos
0	./crash
3.1M	./ctcss-tmp
7.7M	./ctcss
37G	.
[root@cc var]# cd lib/
[root@cc lib]# ls 
alternatives  chrony  containerd  dhclient  docker  initramfs  misc            os-prober  polkit-1   private  rpm-state  selinux  systemd  tuned    up2date
authselect    cloud   dbus        dnf       games   logrotate  NetworkManager  plymouth   portables  rpm      rsyslog    sss      tpm      unbound
[root@cc lib]# du -h --max-depth=1
4.4M	./dnf
55M	./rpm
0	./games
0	./misc
0	./rpm-state
60K	./alternatives
27M	./selinux
0	./dbus
0	./initramfs
0	./os-prober
136K	./systemd
0	./portables
0	./private
0	./tpm
0	./polkit-1
16K	./NetworkManager
4.0K	./unbound
56K	./authselect
4.0K	./dhclient
4.0K	./plymouth
4.0K	./logrotate
0	./up2date
18M	./sss
0	./tuned
4.0K	./rsyslog
4.0K	./chrony
172K	./cloud
344K	./containerd
36G	./docker
36G	.
[root@cc lib]# docker ps 
CONTAINER ID   IMAGE          COMMAND                  CREATED        STATUS        PORTS                                                  NAMES
252578bb1380   2fe463762680   "docker-entrypoint.s…"   9 months ago   Up 6 months   33060/tcp, 0.0.0.0:3307->3306/tcp, :::3307->3306/tcp   mysql
[root@cc lib]# cd docker/
[root@cc docker]# ls 
buildkit  containers  image  network  overlay2  plugins  runtimes  swarm  tmp  trust  volumes
[root@cc docker]# du -h --max-depth=1
33G	./containers
0	./plugins
2.7G	./overlay2
4.5M	./image
388M	./volumes
0	./trust
72K	./network
0	./swarm
72K	./buildkit
0	./tmp
0	./runtimes
36G	.
[root@cc docker]# cd containers/
[root@cc containers]# l 
-bash: l: command not found
[root@cc containers]# ls 
252578bb1380d0bdbddd889e0386c4810cc793ec44376386e48cf0a820b3bf8f  a618ddd909305d56e4d82fb32f6aee3cdda338acaaf7429be9fa21a44b01e8e9
[root@cc containers]# du -h --max-depth=1
236K	./252578bb1380d0bdbddd889e0386c4810cc793ec44376386e48cf0a820b3bf8f
33G	./a618ddd909305d56e4d82fb32f6aee3cdda338acaaf7429be9fa21a44b01e8e9
33G	.

最后可以看到和Docker容器有关，通过docker ps -a 查看我就安装了一个Mysql和mongo(未运行) 容器。

[root@cc a618ddd909305d56e4d82fb32f6aee3cdda338acaaf7429be9fa21a44b01e8e9]# du -h --max-depth=1
0	./checkpoints
0	./mounts
33G	.
[root@cc a618ddd909305d56e4d82fb32f6aee3cdda338acaaf7429be9fa21a44b01e8e9]# docker ps 
CONTAINER ID   IMAGE          COMMAND                  CREATED        STATUS        PORTS                                                  NAMES
252578bb1380   2fe463762680   "docker-entrypoint.s…"   9 months ago   Up 6 months   33060/tcp, 0.0.0.0:3307->3306/tcp, :::3307->3306/tcp   mysql
[root@cc a618ddd909305d56e4d82fb32f6aee3cdda338acaaf7429be9fa21a44b01e8e9]# docker ps -a 
CONTAINER ID   IMAGE          COMMAND                  CREATED        STATUS                    PORTS                                                  NAMES
a618ddd90930   mongo:latest   "docker-entrypoint.s…"   6 months ago   Exited (14) 3 weeks ago                                                          mongo-copy-one
252578bb1380   2fe463762680   "docker-entrypoint.s…"   9 months ago   Up 6 months               33060/tcp, 0.0.0.0:3307->3306/tcp, :::3307->3306/tcp   mysql

经过对比发现是Mongo容器磁盘占用满了

[root@cc a618ddd909305d56e4d82fb32f6aee3cdda338acaaf7429be9fa21a44b01e8e9]# ll -h
total 33G
-rw-r----- 1 root root  33G Jul 25 15:37 a618ddd909305d56e4d82fb32f6aee3cdda338acaaf7429be9fa21a44b01e8e9-json.log
drwx------ 2 root root    6 Jan 21  2022 checkpoints
-rw------- 1 root root 3.5K Jul 25 15:33 config.v2.json
-rw-r--r-- 1 root root 1.6K Jul 25 15:33 hostconfig.json
-rw-r--r-- 1 root root   13 Jul 25 15:33 hostname
-rw-r--r-- 1 root root  174 Jul 25 15:33 hosts
drwx-----x 2 root root    6 Jan 21  2022 mounts
-rw-r--r-- 1 root root  117 Jul 25 15:33 resolv.conf
-rw-r--r-- 1 root root   71 Jul 25 15:33 resolv.conf.hash

可以看到光日志文件就占了33G；

解决方式

MongoDB的日志文件在设置 logappend=true 的情况下，会不断向同一日志文件追加的，时间长了，自然变得非常大。

解决如下：（特别注意：启动的时候必须是–logpath指定了log路径的）

用mongo连接到服务端

复制代码代码如下:

use admin //切换到admin数据库
db.runCommand({logRotate:1})
这样会使mongo关闭当前日志文件，重启一个新的日志文件，不需要停止mongodb服务。

docker exec -it a618ddd90930 mongo admin
MongoDB shell version v5.0.5
connecting to: mongodb://127.0.0.1:27017/admin?compressors=disabled&gssapiServiceName=mongodb
Implicit session: session { "id" : UUID("48a99cf7-813c-4c3d-ad02-68d138321e63") }
MongoDB server version: 5.0.5
================
Warning: the "mongo" shell has been superseded by "mongosh",
which delivers improved usability and compatibility.The "mongo" shell has been deprecated and will be removed in
an upcoming release.
For installation instructions, see
https://docs.mongodb.com/mongodb-shell/install/
================
---
The server generated these startup warnings when booting: 
        2022-07-25T07:34:01.326+00:00: Access control is not enabled for the database. Read and write access to data and configuration is unrestricted
        2022-07-25T07:34:01.326+00:00: /sys/kernel/mm/transparent_hugepage/enabled is 'always'. We suggest setting it to 'never'
---
---
        Enable MongoDB's free cloud-based monitoring service, which will then receive and display
        metrics about your deployment (disk utilization, CPU, operation statistics, etc).

        The monitoring data will be available on a MongoDB website with a unique URL accessible to you
        and anyone you share the URL with. MongoDB may use this information to make product
        improvements and to suggest MongoDB products and deployment options to you.

        To enable free monitoring, run the following command: db.enableFreeMonitoring()
        To permanently disable this reminder, run the following command: db.disableFreeMonitoring()
---
copySet:SECONDARY> use admin  //切换到admin数据库
Error: [admin  //切换到admin数据库] is not a valid database name :
Mongo.prototype.getDB@src/mongo/shell/mongo.js:61:12
getDatabase@src/mongo/shell/session.js:933:28
DB.prototype.getSiblingDB@src/mongo/shell/db.js:30:12
shellHelper.use@src/mongo/shell/utils.js:851:10
shellHelper@src/mongo/shell/utils.js:838:15
@(shellhelp2):1:1
copySet:SECONDARY> use admin
switched to db admin
copySet:SECONDARY> db.runCommand({logRotate:1})
{
	"ok" : 1,
	"$clusterTime" : {
		"clusterTime" : Timestamp(1643352063, 1),
		"signature" : {
			"hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),
			"keyId" : NumberLong(0)
		}
	},
	"operationTime" : Timestamp(1643352063, 1)
}
copySet:SECONDARY>