Last active
April 28, 2017 08:10
-
-
Save nikhilbansal/f49ab0eeeaa2fb8634d6d35ed475c3ea to your computer and use it in GitHub Desktop.
Linux df and du mismatching
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
The df -h on the partition /data was showing full usage of 20GB but du -sh /data was showing a usage of only 581MB. | |
ubuntu@ip-xxx-xx-x-xx:/data$ df -h | |
Filesystem Size Used Avail Use% Mounted on | |
... | |
/dev/xvdb 20G 20G 0G 100% /data | |
ubuntu@ip-xxx-xx-x-xx:/data$ du -sh . | |
581M . | |
This huge difference means that files are deleted from filesystem but are still referenced from some open applications. | |
There were no results of deleted files after running the following command. | |
ubuntu@ip-xxx-xx-x-xx:/data$ lsof -nP | grep '(deleted)' | |
ubuntu@ip-xxx-xx-x-xx:/data$ | |
After few minutes of confusion I tried running the same command with sudo and there it was. | |
ubuntu@ip-xxx-xx-x-xx:/data$ sudo lsof -nP | grep deleted | |
sh 5267 root cwd DIR 202,16 0 393217 /data/xxxxxxxxxxxxx (deleted) | |
sh 5267 root 2w REG 202,16 3742909615 397337 /data/xxxxxxxxxxxxx/nohup.out (deleted) | |
tail 5270 root cwd DIR 202,16 0 393217 /data/xxxxxxxxxxxxx (deleted) | |
tail 5270 root 2w REG 202,16 3742909615 397337 /data/xxxxxxxxxxxxx/nohup.out (deleted) | |
tail 5270 root 3r REG 202,16 1196962556 397656 /data/xxxxxxxxxxxxx/log/production.log (deleted) | |
tail 14934 root 2w REG 202,16 1424371 398138 /data/xxxxxxxxxxxxx/nohup.out~ (deleted) | |
tail 14934 root 3r REG 202,16 9281997539 659099 /data/xxxxxxxxxxxxx/log/production.log.1 (deleted) | |
tail 19409 root 2w REG 202,16 1424371 398138 /data/xxxxxxxxxxxxx/nohup.out~ (deleted) | |
tail 19409 root 3r REG 202,16 4805248107 655668 /data/xxxxxxxxxxxxx/log/production.log.9 (deleted) | |
Looking at the corresponding processes, I found that there were open processes that were holding references to the old files. | |
ubuntu@ip-xxx-xx-x-xx:/data$ ps -ef | grep 5267 | |
ubuntu 2712 1616 0 13:13 pts/3 00:00:00 grep --color=auto 5267 | |
root 5267 1 0 Jan10 ? 00:00:00 sh -c tail -f -n 0 "/data/xxxxxxxxxxxxx/log/production.log" | |
root 5270 5267 0 Jan10 ? 00:00:00 tail -f -n 0 /data/xxxxxxxxxxxxx/log/production.log | |
After killing all of such process, df and du matched completely and 18GB of disk space got freed up. | |
ubuntu@ip-xxx-xx-x-xx:/data$ df -h | |
Filesystem Size Used Avail Use% Mounted on | |
... | |
/dev/xvdb 20G 625M 18G 4% /data | |
Hurray! | |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment