Tracking down where disk space has gone on Linux?

797,833

Solution 1

Don't go straight to du /. Use df to find the partition that's hurting you, and then try du commands.

One I like to try is

# U.S.
du -h <dir> | grep '[0-9\.]\+G'
# Others
du -h <dir> | grep '[0-9\,]\+G'

because it prints sizes in "human readable form". Unless you've got really small partitions, grepping for directories in the gigabytes is a pretty good filter for what you want. This will take you some time, but unless you have quotas set up, I think that's just the way it's going to be.

As @jchavannes points out in the comments, the expression can get more precise if you're finding too many false positives. I incorporated the suggestion, which does make it better, but there are still false positives, so there are just tradeoffs (simpler expr, worse results; more complex and longer expr, better results). If you have too many little directories showing up in your output, adjust your regex accordingly. For example,

grep '^\s*[0-9\.]\+G'

is even more accurate (no < 1GB directories will be listed).

If you do have quotas, you can use

quota -v

to find users that are hogging the disk.

Solution 2

For a first look, use the “summary” view of du:

du -s /*

The effect is to print the size of each of its arguments, i.e. every root folder in the case above.

Furthermore, both GNU du and BSD du can be depth-restricted (but POSIX du cannot!):

  • GNU (Linux, …):

    du --max-depth 3
    
  • BSD (macOS, …):

    du -d 3
    

This will limit the output display to depth 3. The calculated and displayed size is still the total of the full depth, of course. But despite this, restricting the display depth drastically speeds up the calculation.

Another helpful option is -h (words on both GNU and BSD but, once again, not on POSIX-only du) for “human-readable” output (i.e. using KiB, MiB etc.).

Solution 3

You can also run the following command using du:

~# du -Pshx /* 2>/dev/null
  • The -s option summarizes and displays total for each argument.
  • -h prints Mio, Gio, etc.
  • -x = stay in one filesystem (very useful).
  • -P = don't follow symlinks (which could cause files to be counted twice for instance).

Be careful with -x, which will not show the /root directory if it is on a different filesystem. In that case, you have to run du -Pshx /root 2>/dev/null to show it (once, I struggled a lot not pointing out that my /root directory had gone full).

Solution 4

Finding the biggest files on the filesystem is always going to take a long time. By definition you have to traverse the whole filesystem looking for big files. The only solution is probably to run a cron job on all your systems to have the file ready ahead of time.

One other thing, the x option of du is useful to keep du from following mount points into other filesystems. I.e:

du -x [path]

The full command I usually run is:

sudo du -xm / | sort -rn > usage.txt

The -m means return results in megabytes, and sort -rn will sort the results largest number first. You can then open usage.txt in an editor, and the biggest folders (starting with /) will be at the top.

Solution 5

I always use du -sm * | sort -n, which gives you a sorted list of how much the subdirectories of the current working directory use up, in mebibytes.

You can also try Konqueror, which has a "size view" mode, which is similar to what WinDirStat does on Windows: it gives you a viual representation of which files/directories use up most of your space.

Update: on more recent versions, you can also use du -sh * | sort -h which will show human-readable filesizes and sort by those. (numbers will be suffixed with K, M, G, ...)

For people looking for an alternative to KDE3's Konqueror file size view may take a look at filelight, though it's not quite as nice.

Share:
797,833

Related videos on Youtube

Admin
Author by

Admin

Updated on June 15, 2022

Comments

  • Admin
    Admin less than a minute

    When administering Linux systems I often find myself struggling to track down the culprit after a partition goes full. I normally use du / | sort -nr but on a large filesystem this takes a long time before any results are returned.

    Also, this is usually successful in highlighting the worst offender but I've often found myself resorting to du without the sort in more subtle cases and then had to trawl through the output.

    I'd prefer a command line solution which relies on standard Linux commands since I have to administer quite a few systems and installing new software is a hassle (especially when out of disk space!)

  • Admin
    Admin over 13 years
    That's only Konqueror 3.x though - the file size view still hasn't been ported to KDE4.
  • SamB
    SamB about 12 years
    Thanks for pointing out the -x flag!
  • zzapper
    zzapper over 9 years
    This is very quick, simple and practical
  • thegreendroid
    thegreendroid almost 9 years
    This command did the trick to find a hidden folder that seemed to be increasing in size over time. Thanks!
  • ReactiveRaven
    ReactiveRaven almost 9 years
    if du complains about -d try --max-depth 5 in stead.
  • Admin
    Admin over 8 years
    Great anwser. Seems correct for me. I suggest du -hcd 1 /directory. -h for human readable, c for total and d for depth.
  • jchavannes
    jchavannes almost 8 years
    grep '[0-9]G' contained a lot of false positives and also omitted any decimals. This worked better for me: sudo du -h / | grep -P '^[0-9\.]+G'
  • jchavannes
    jchavannes almost 8 years
    @BenCollins I think you also need the -P flag for Perl regex.
  • Ben Collins
    Ben Collins almost 8 years
    @jchavannes -P is unnecessary for this expression because there's nothing specific to Perl there. Also, -P isn't portable to systems that don't have the GNU implementation.
  • jchavannes
    jchavannes almost 8 years
    Ahh. Well having a carat at the beginning will remove false positives of directories which have a number followed by a G in the name, which I did.
  • User
    User over 7 years
    Is this in bytes?
  • serg10
    serg10 over 7 years
    By default, on my system, 'du -S' gives a nice human readable output. You get a plain number of bytes for small files, then a number with a 'KB' or 'MB' suffix for bigger files.
  • Vitruvie
    Vitruvie about 7 years
    In case you have really big directories, you'll want [GT] instead of just G
  • CMCDragonkai
    CMCDragonkai about 7 years
    Is there a tool that will continuously monitor disk usage across all directories (lazily) in the filesystem? Something that can be streamed to a web UI? Preferably soft-realtime information.
  • rjmunro
    rjmunro almost 7 years
    You can use the -x option of find to make sure you don't find files on other devices than the start point of your find command. This fixes the remotely mounted drives issue.
  • Siddhartha
    Siddhartha over 6 years
    You could do du -Sh to get a human readable output.
  • Mike
    Mike over 6 years
    du -Pshx .* * 2>/dev/null + hidden/system directories
  • augurar
    augurar about 6 years
    I like to use du -h | sort -hr | head
  • ndemou
    ndemou over 5 years
    It's a pity that a dozen of grep hacks are more upvoted. Oh and du -k will make it absolutely certain that du is using KB units
  • Mark Borgerding
    Mark Borgerding over 5 years
    Good idea about the -k. Edited.
  • dave_thompson_085
    dave_thompson_085 over 5 years
    Even simpler and more robust: du -kx $2 | awk '$1>'$(($1*1024)) (if you specify only a condition aka pattern to awk the default action is print $0)
  • ndemou
    ndemou over 5 years
    Good point @date_thompson_085. That's true for all versions of awk I know of (net/free-BSD & GNU). @mark-borgerding so this means that you can greatly simplify your first example to just du -kx / | awk '$1 > 500000'
  • ndemou
    ndemou over 5 years
    @mark-borgerding: If you have just a few kBytes left somewhere you can also keep the whole output of du like this du -kx / | tee /tmp/du.log | awk '$1 > 500000'. This is very helpful because if your first filtering turns out to be fruitless you can try other values like this awk '$1 > 200000' /tmp/du.log or inspect the complete output like this sort -nr /tmp/du.log|less without re-scanning the whole filesystem
  • Mark Borgerding
    Mark Borgerding over 5 years
    Regarding the simplification -- I think that kills clarity to save a few characters.
  • Mark Borgerding
    Mark Borgerding over 5 years
    Regarding saving the whole du output, -- That "few kBytes" could easily be many megabytes if the volume contains millions of files. That seems dangerous under the presumable circumstances.
  • jonathanccalixto
    jonathanccalixto over 5 years
    I'm use du -hd 1 <folder to inspect> | sort -hr | head
  • pahariayogi
    pahariayogi about 5 years
    'du -sh * | sort -h ' works perfectly on my Linux (Centos distro) box. Thanks!
  • sourcejedi
    sourcejedi about 5 years
    Using du -s means this will print a total size for / and nothing else.
  • ndemou
    ndemou about 5 years
    This tool doesn't match two main points of the question "I often find myself struggling to track down the culprit after a partition goes full" and "I'd prefer a command line solution which relies on standard Linux commands"
  • ndemou
    ndemou about 5 years
    Good point but this should be a comment and not an answer by itself - this question suffers from too many answers
  • ndemou
    ndemou almost 5 years
    Note this part of the question: "I'd prefer a command line solution which relies on standard Linux commands since..."
  • ndemou
    ndemou almost 5 years
    Don't use grep for arithmetic operations -- use awk instead: du -k | awk '$1 > 500000'. It is much easier to understand, edit and get correct on the first try.
  • srghma
    srghma almost 5 years
    du --max-depth 5 -h /* 2>&1 | grep '[0-9\.]\+G' | sort -hr | head to filter Permission denied
  • B. Shea
    B. Shea over 4 years
    "finding biggest takes long time.." -> Well it depends, but tend to disagree: doesn't take that long with utilities like ncdu - at least quicker than du or find (depending on depth and arguments)..
  • vimal krishna
    vimal krishna over 4 years
    I find this to be the best answer, to detect the large sized in sorted order
  • Clare Macrae
    Clare Macrae over 4 years
    @Siddhartha If you add -h, it will likely change the effect of the sort -nr command - meaning the sort will no longer work, and then the head command will also no longer work
  • oarfish
    oarfish almost 4 years
    On Ubuntu, I need to use -h to du for human readable numbers, as well as sort -h for human-numeric sort. The list is sorted in reverse, so either use tail or change order.
  • Bruno
    Bruno over 3 years
    since I prefer not to be root, I had to adapt where the file is written : sudo du -xm / | sort -rn > ~/usage.txt
  • Admin
    Admin almost 3 years
    If you can't use a GUI (like you're on a remote server), ncdu -e works nicely. Once the display opens up, use m then M to display and sort by mtime, while the (admittedly small) percentage graph is still there to get you an idea of the size.
  • symcbean
    symcbean almost 3 years
    "If you can't use a GUI (like you're on a remote server)," - why does a remote server prevent you from using a gui?
  • Atralb
    Atralb over 2 years
    /root/shows without issues. Why would it not be shown ?
  • Dennis
    Dennis over 1 year
    ncdu -e is wrong becasue it requires an argument
  • WBT
    WBT over 1 year
    Can't run Java on Linux?
  • Faither
    Faither over 1 year
    The OP asked for a CLI version
  • WBT
    WBT over 1 year
    OP said that was preferred, not required; questions closed as duplicates of this don't have the same preference.