根目录100%使用率

mssdfdsgd:~ # lsof +L1
COMMAND     PID USER   FD   TYPE DEVICE    SIZE/OFF NLINK    NODE NAME
tcpdump    3543 root    1w   REG    8,6          88     0 2083099 /root/nohup.out (deleted)
tcpdump    3543 root    2w   REG    8,6          88     0 2083099 /root/nohup.out (deleted)
tcpdump    3543 root    4w   REG    8,6 36285661184     0 2083100 /root/tcpdump.out (deleted)
mysqld_sa  9441 root    1u   CHR  136,0         0t0     0       3 /dev/pts/0 (deleted)
mysqld_sa  9441 root    2u   CHR  136,0         0t0     0       3 /dev/pts/0 (deleted)
 

 

 

Removing log files (or any other files) that are still "open" means that only the name is deleted. The file contents are stored in the inode which won't go away until the process that had the file open stops. I'd try stopping syslogd to see if that helps with the file(s) you arleady deleted. (Also if you're running sendmail it might be writing to /var/log/mail*log. Other processes may be writing to other logs.)

If /var is part of / as it appears to be on your system you should focus on it. /var/log is a good starting point. /var/tmp is another. (In fact here recently we found our Oracle DB apparently had something writing something huge to /var/tmp that caused issues.)
If you do a cd /var then run "du -sk |sort -n" it will show the largest items (files/directories) at the bottom of the output. Check out the largest. (If a directory you can do the same du -sk |sort -n to drill down on its largest.)

 

Lots of things log or write to /var and it isn't all in /var/log. You don't mention what distro you're on. On RHEL5/CentOS I have often seen /var/cache/yum fill up. (There is a newer yum that fixes this issue.) You can run "yum clean all" to clear that directory. /var/wtmp which stores user logins can get rather large. On Dell PowerEdge Systems with Dell OpenManage installed I've seen large /var/log/TTY* files that can be cleaned but as noted above they are "open" so you have to first shutdown openmanage before clearing them. These are just a few examples.
After you've resolved the issue you might want to consider creating /var as a separate mount (of course you'll need downtime to do that).

 

As I noted in my earlier post when you delete files that are "open" only the names are deleted. The actual space of files and the inodes are NOT deleted. Stopping the processes that have the files open will close the files and allow the deletion to complete.

If you reboot the server as you did then you have stopped ALL processes including those that had files open and thereby resolved the issue.

The moral of the story: Do NOT delete log files or other files that are written to frequently - it is better to do ">file" which will truncate the file to 0 bytes.

Also you can search for processes that have files open with no name by using the lsof command and looking for large CHR files in its output that have no name. The PID will be the process that has the file open and killing the process will free the space.

 

 

The /proc directory is a pseudo filesystem. The files are created by the kernel when you read them. It isn't taking up GBs of space.

One thing to look at is deleted files which are still open. Restarting a service may clear that up. This is done to prevent name collision, and prevent a stale file from remaining in the event the service crashed.

Another user had a problem with a new and an old instance of a java service running. Each had a deleted file open around 10GB of size.

You can detect open deleted files with: lsof +L1

Another thing could be files in /tmp and /var/tmp taking up a lot of space. Your server may be configured to delete files in /tmp and /var/tmp during a reboot.

---
The root partition is very small to begin with. Your /tmp and /var/tmp are almost 3 times as large.
 

 

 

 

If this is a new install, I would advice a fresh installation with the bulk of the space taken for where the database is contained. I imagine this would need much more space when it goes online or has been used for a while.

For example, for a mysql database, the /var/ directory would probably contain the database. Some people create a separate partition for both /var and /var/log/ on servers. The later is to prevent an attacker from filling up the logs and depriving the root partition of disk space. On my distribution, mysql uses /var/lib/ while other services such as apached tend to use /srv/. It may depend on which database you are using and even the version.

If this is a server, does it use raid for redundancy?

Your post is in Linux Server but it seems to be for a Solaris machine. There will be some difference for Solaris such as the partition names, but many of the principles are the same. On a Linux server, world writable directories should be mounted on a partition that used the noexec and nodev mount options. If you have these same mount options, you may want the partition with /tmp to be mounted this way as well.

I still don't understand the partitions directories with the names of libraries. Did you do that manually? Or is that a Solaris thing. It almost looks hackey to my eyes.