администрирование – Freeing up space with a deleted file

Question:

A situation has arisen with the following conditions:

server with ubuntu distribution
problem with atop , due to which he wrote four hundred megabytes to /run/atop/atop.acct I don’t know what (I didn’t deal with atop before and accepted the server already with it; as far as I understand, it costs because of munin ). The file itself was killed via rm .
completely clogged /run (df shows 100%, du shows significantly less usage by the files present in the listing)
/run/atop/atop.acct removed via rm and is not present in the physical file listing nor in the output lsof / lsof +L1
/run/atop.pid exists but is empty
there is nothing like atop in the output of ps aux
Writing new files to /run is not possible due to lack of space

Output df -h :

 Filesystem Size Used Avail Use% Mounted on ... tmpfs 396M 396M 0 100% /run ...

du -ach /run/* | grep atop :

 0 /run/atop/atop.acct 0 /run/atop 0 /run/atop.pid

/run is located on tmpfs, not mounted elsewhere, /var/run is symlinked to it:

 tmpfs on /run type tmpfs (rw,noexec,nosuid,size=10%,mode=0755) none on /run/lock type tmpfs (rw,noexec,nosuid,nodev,size=5242880) none on /run/shm type tmpfs (rw,nosuid,nodev) none on /run/user type tmpfs (rw,noexec,nosuid,nodev,size=104857600,mode=0755)

lsof | head -1 && lsof | grep /run :

 COMMAND PID TID USER FD TYPE DEVICE SIZE/OFF NODE NAME systemd-u 405 root 3u unix 0xffff880036278000 0t0 12416 /run/udev/control dbus-daem 1661 messagebus 4u unix 0xffff880036262940 0t0 8037 /var/run/dbus/system_bus_socket dbus-daem 1661 messagebus 8u unix 0xffff8800369d8f00 0t0 1608 /var/run/dbus/system_bus_socket dbus-daem 1661 messagebus 9u unix 0xffff880036263840 0t0 10865 /var/run/dbus/system_bus_socket systemd-l 1744 root 13r FIFO 0,16 0t0 17443 /run/systemd/sessions/c1.ref systemd-l 1744 root 14r FIFO 0,16 0t0 142943477 /run/systemd/sessions/20.ref systemd-l 1744 root 15r FIFO 0,16 0t0 144221305 /run/systemd/sessions/23.ref zabbix_ag 1818 zabbix 3wW REG 0,16 4 11548 /run/zabbix/zabbix_agentd.pid zabbix_ag 1830 zabbix 3w REG 0,16 4 11548 /run/zabbix/zabbix_agentd.pid zabbix_ag 1831 zabbix 3w REG 0,16 4 11548 /run/zabbix/zabbix_agentd.pid zabbix_ag 1832 zabbix 3w REG 0,16 4 11548 /run/zabbix/zabbix_agentd.pid zabbix_ag 1836 zabbix 3w REG 0,16 4 11548 /run/zabbix/zabbix_agentd.pid zabbix_ag 1837 zabbix 3w REG 0,16 4 11548 /run/zabbix/zabbix_agentd.pid cron 2049 root 3u REG 0,16 5 8102 /run/crond.pid apache2 2305 www-data 15w REG 0,19 0 143142467 /run/lock/apache2/fcgid-proctbl.3278 (deleted) apache2 2305 www-data 20w REG 0,19 0 143142470 /run/lock/apache2/fcgid-pipe.3278 (deleted) apache2 2305 www-data 22w REG 0,19 0 143142473 /run/lock/apache2/rewrite-map.3278 (deleted) apache2 2305 www-data 23w REG 0,19 0 143142474 /run/lock/apache2/mpm-accept.3278 (deleted) apache2 2701 www-data 15w REG 0,19 0 143142467 /run/lock/apache2/fcgid-proctbl.3278 (deleted) apache2 2701 www-data 20w REG 0,19 0 143142470 /run/lock/apache2/fcgid-pipe.3278 (deleted) apache2 2701 www-data 22w REG 0,19 0 143142473 /run/lock/apache2/rewrite-map.3278 (deleted) apache2 2701 www-data 23w REG 0,19 0 143142474 /run/lock/apache2/mpm-accept.3278 (deleted) apache2 3238 www-data 15w REG 0,19 0 143142467 /run/lock/apache2/fcgid-proctbl.3278 (deleted) apache2 3238 www-data 20w REG 0,19 0 143142470 /run/lock/apache2/fcgid-pipe.3278 (deleted) apache2 3238 www-data 22w REG 0,19 0 143142473 /run/lock/apache2/rewrite-map.3278 (deleted) apache2 3238 www-data 23w REG 0,19 0 143142474 /run/lock/apache2/mpm-accept.3278 (deleted) apache2 3278 root 15w REG 0,19 0 143142467 /run/lock/apache2/fcgid-proctbl.3278 (deleted) apache2 3278 root 20w REG 0,19 0 143142470 /run/lock/apache2/fcgid-pipe.3278 (deleted) apache2 3278 root 22w REG 0,19 0 143142473 /run/lock/apache2/rewrite-map.3278 (deleted) apache2 3278 root 23w REG 0,19 0 143142474 /run/lock/apache2/mpm-accept.3278 (deleted) login 3319 root 6w FIFO 0,16 0t0 17443 /run/systemd/sessions/c1.ref apache2 6811 www-data 15w REG 0,19 0 143142467 /run/lock/apache2/fcgid-proctbl.3278 (deleted) apache2 6811 www-data 20w REG 0,19 0 143142470 /run/lock/apache2/fcgid-pipe.3278 (deleted) apache2 6811 www-data 22w REG 0,19 0 143142473 /run/lock/apache2/rewrite-map.3278 (deleted) apache2 6811 www-data 23w REG 0,19 0 143142474 /run/lock/apache2/mpm-accept.3278 (deleted) apache2 11213 www-data 15w REG 0,19 0 143142467 /run/lock/apache2/fcgid-proctbl.3278 (deleted) apache2 11213 www-data 20w REG 0,19 0 143142470 /run/lock/apache2/fcgid-pipe.3278 (deleted) apache2 14283 www-data 15w REG 0,19 0 143142467 /run/lock/apache2/fcgid-proctbl.3278 (deleted) apache2 14283 www-data 20w REG 0,19 0 143142470 /run/lock/apache2/fcgid-pipe.3278 (deleted) apache2 14283 www-data 22w REG 0,19 0 143142473 /run/lock/apache2/rewrite-map.3278 (deleted) apache2 14283 www-data 23w REG 0,19 0 143142474 /run/lock/apache2/mpm-accept.3278 (deleted) apache2 17302 www-data 15w REG 0,19 0 143142467 /run/lock/apache2/fcgid-proctbl.3278 (deleted) apache2 17302 www-data 20w REG 0,19 0 143142470 /run/lock/apache2/fcgid-pipe.3278 (deleted) apache2 17302 www-data 22w REG 0,19 0 143142473 /run/lock/apache2/rewrite-map.3278 (deleted) apache2 17302 www-data 23w REG 0,19 0 143142474 /run/lock/apache2/mpm-accept.3278 (deleted) apache2 17303 www-data 15w REG 0,19 0 143142467 /run/lock/apache2/fcgid-proctbl.3278 (deleted) apache2 17303 www-data 20w REG 0,19 0 143142470 /run/lock/apache2/fcgid-pipe.3278 (deleted) apache2 17303 www-data 22w REG 0,19 0 143142473 /run/lock/apache2/rewrite-map.3278 (deleted) apache2 17303 www-data 23w REG 0,19 0 143142474 /run/lock/apache2/mpm-accept.3278 (deleted) apache2 19889 www-data 15w REG 0,19 0 143142467 /run/lock/apache2/fcgid-proctbl.3278 (deleted) apache2 19889 www-data 20w REG 0,19 0 143142470 /run/lock/apache2/fcgid-pipe.3278 (deleted) apache2 19889 www-data 22w REG 0,19 0 143142473 /run/lock/apache2/rewrite-map.3278 (deleted) apache2 19889 www-data 23w REG 0,19 0 143142474 /run/lock/apache2/mpm-accept.3278 (deleted) apache2 21403 www-data 15w REG 0,19 0 143142467 /run/lock/apache2/fcgid-proctbl.3278 (deleted) apache2 21403 www-data 20w REG 0,19 0 143142470 /run/lock/apache2/fcgid-pipe.3278 (deleted) apache2 21403 www-data 22w REG 0,19 0 143142473 /run/lock/apache2/rewrite-map.3278 (deleted) apache2 21403 www-data 23w REG 0,19 0 143142474 /run/lock/apache2/mpm-accept.3278 (deleted) sshd 21758 root 7w FIFO 0,16 0t0 142943477 /run/systemd/sessions/20.ref apache2 22095 www-data 15w REG 0,19 0 143142467 /run/lock/apache2/fcgid-proctbl.3278 (deleted) apache2 22095 www-data 20w REG 0,19 0 143142470 /run/lock/apache2/fcgid-pipe.3278 (deleted) apache2 22095 www-data 22w REG 0,19 0 143142473 /run/lock/apache2/rewrite-map.3278 (deleted) apache2 22095 www-data 23w REG 0,19 0 143142474 /run/lock/apache2/mpm-accept.3278 (deleted) sshd 22248 root 7w FIFO 0,16 0t0 144221305 /run/systemd/sessions/23.ref

How do I clean up /run in this situation?

Answer:

The problem here is that open files are not deleted, but continue to exist while the program that opened them is running.

You can verify this by opening the file for writing in one console:

$ cat > /tmp/example.txt

And then from another console deleting this file:

$ ls /tmp/example.txt 
/tmp/example.txt
$ rm /tmp/example.txt

Then you can check that the file still exists:

$ lsof -a +L1 /tmp
COMMAND     PID  USER   FD   TYPE DEVICE SIZE/OFF NLINK     NODE NAME
cat       27223 sanmai    1w   REG   0,47       25     0 23585409 /tmp/example.txt (deleted)

One solution would be to either close or restart the program holding the deleted file open. But usually programs support handling the HUP signal precisely for the purpose of reopening log files and other reinitialization, atop is no exception (only this program does not restart itself):

$ sudo pkill -HUP atop

Therefore, it will be a more correct way to solve this problem.

Post Views: 1