In this post, we will try to see those cases where we faced disk space issues where we can’t find files or directories consuming space, so after searching around space consuming files, we don’t fond nothing wither in file stored in Disk and Open files this is really annoying sometime. Same situation happened in real scenario where it was really difficult to find real cause of it.

In various situations causes could different but in this i am going to talk about any partition space which get hide through some mount point.Like, there is directory /data inside / mount point which has some data that got hide if we mount another partition on /data.

So let’s try to explain this situation with creating it and further try to troubleshoot it, in a way we could able find space consuming files and free up some space.

SetUP

In this setup, i am using CentOS 6.10

[root@srv1 ~]# cat /etc/redhat-release 
CentOS release 6.10 (Final)
[root@srv1 ~]# uname -r
2.6.32-754.10.1.el6.x86_64

Issues

  • Unable find space consume in a partition
  • du and df command has difference size of partition
  • Hence we don’t find file consuming space, we can’t make it free, which cause it fill up completely

So we have / partition, which has around 2.1 GB of space like below

[root@srv1 ~]# df -hTP .
Filesystem                   Type  Size  Used Avail Use% Mounted on
/dev/mapper/VolGroup-lv_root ext4   18G  2.1G   15G  13% /

Now let’s create one directory to carry a file that has some noticeable space on same partition.

[root@srv1 ~]# mkdir /data
[root@srv1 ~]# cd /data/
[root@srv1 data]# ls
[root@srv1 data]# df -hTP .
Filesystem                   Type  Size  Used Avail Use% Mounted on
/dev/mapper/VolGroup-lv_root ext4   18G  2.1G   15G  13% /

Let’s create one file

[root@srv1 data]# dd if=/dev/zero of=/data/dd_file bs=1M count=4096
4096+0 records in
4096+0 records out
4294967296 bytes (4.3 GB) copied, 23.1992 s, 185 MB/s
[root@srv1 data]# ll
total 4194308
-rw-r--r-- 1 root root 4294967296 Mar  9 19:46 dd_file

So above i created one directory name /data and create 4GB of file within same directory.

Now i am going to mount another partition on same /data directory which going to hide this 4 GB file, which taking space from / partition.

[root@srv1 ~]# mount /dev/vdb1 /data
EXT4-fs (vdb1): mounted filesystem with ordered data mode. Opts: 

[root@srv1 ~]# cd /data/
[root@srv1 data]# ls
lost+found
[root@srv1 data]# df -hTl .
Filesystem     Type  Size  Used Avail Use% Mounted on
/dev/vdb1      ext4   14M  134K   13M   2% /data

Now you see, as we mount another partition on /data, that file is got hide.This way, we have 4GB of / partition is kept hide in Disk, but there is no way to use or work on same. But what if you are having this issues in production setup where you can’t unmount any partitions to clean up this space.

For such case we have one options through which we could access that hide file which should cleanup, this option is called mounted over which basically work like mount partition that contain file that consuming space, like below.

[root@srv1 ~]# mkdir /tmp/root_tmp

[root@srv1 ~]# mount --bind / /tmp/root_tmp/
[root@srv1 ~]# cd /tmp/root_tmp/
[root@srv1 root_tmp]# ls
bin  boot  data  dev  etc  home  lib  lib64  lost+found  media	mnt  opt  proc	root  sbin  selinux  srv  sys  tmp  usr  var

[root@srv1 root_tmp]# cd data/
[root@srv1 data]# ls
dd_file

So we mount / again and now this time data directory contain that big file that we need to delete to free up space from filled up mount point, this method is completely safe while online, we don’t need to reboot or umount partition.

Again, see which mount where

[root@srv1 data]# pwd  #mount / again 
/tmp/root_tmp/data
[root@srv1 data]# ls -lh
total 4.1G
-rw-r--r-- 1 root root 4.0G Mar  9 19:46 dd_file #contain dd file 

[root@srv1 data]# ls -lh /data/ #/data mount point
total 12K                       #Nothing much in there
-rw-r--r-- 1 root root   0 Mar 10 02:51 f1
drwx------ 2 root root 12K Mar  9 19:14 lost+found

Now going to delete dd file

[root@srv1 data]# pwd
/tmp/root_tmp/data
[root@srv1 data]# rm -rf dd_file 
[root@srv1 data]# df -hTP .
Filesystem                   Type  Size  Used Avail Use% Mounted on
/dev/mapper/VolGroup-lv_root ext4   18G  2.3G   15G  14% /

I hope it’s clear to everyone that how we find space in partition that hide when mount point of another partition came on directory that contain data.