dstat: a powerful system administration tool

As a system administrator, I'm always looking for new tools to make my job easier. I've been using 'dstat' now for about 3 months, and it's something I don't know how I lived so long without. This tool is quite versatile and informative, and interacts directly with the kernel to give you a nice colorful output of what's happening on any linux machine.

There are a ton of flags that you can pass to dstat to get the data you want.

Note: most of the flags can be combined with eachother (depending on what you're looking for) to have it fill your screen with live data. Be sure you have a wide terminal when combining flags, or dstat will detect it and trim valuable output.

Another favorite which combines a ton of information and is very helpful if you want to relate disk activity to network/CPU usage:

# dstat -tcndylp --top-cpu

(This shows you system time, cpu, net traffic, disk io, load avgs, proc stats, and most expensive CPU process)

Possible internal stats are:

aio, cpu, cpu24, disk, disk24, disk24old, epoch, fs, int, int24, io, ipc, load, lock, mem, net, page,  
page24, proc, raw, socket, swap, swapold, sys, tcp, time, udp, unix, vm

Some of the most useful parts of dstat are actually built in "modules" which also interact directly with the kernel.

The I/O modules are as follows:

     --disk-tps
            per disk transactions per second (tps) stats
       --disk-util
            per disk utilization in percentage
       --top-bio-adv
            show most expensive block I/O process (incl. pid and other stats)
       --top-io-adv
            show most expensive I/O process (incl. pid and other stats)

So, to get a live output of how much disk activity is occurring and which processes are responsible, you could combine all of the above and run:

# dstat --top-io-adv --top-bio-adv --disk-util --disk-tps

Caveats: I/O accounting is only available in kernels >2.6.20.

Another of my favorites which combines a ton of information and is very helpful if you want to relate disk activity to network/CPU usage:

# dstat -tcndylp --top-cpu

(shows you system time, cpu, net traffic, disk io, load avgs, proc stats, and most expensive CPU process)

Here's all the possible flags you can pass to dstat:

      -a, --all
             equals -cdngy (default)
     -c, --cpu
             enable cpu stats (system, user, idle, wait, hardware interrupt, software interrupt)
      -C 0,3,total
             include cpu0, cpu3 and total (when using -c/--cpu)
      -d, --disk
             enable disk stats (read, write)
      -f, --full
             expand -C, -D, -I, -N and -S discovery lists
      -D total,hda
             include total and hda (when using -d/--disk)
      -g, --page
             enable page stats (page in, page out)
      -i, --int
             enable interrupt stats
      -I 5,10
             include interrupt 5 and 10 (when using -i/--int)
      -l, --load
             enable load average stats (1 min, 5 mins, 15mins)
      -m, --mem
             enable memory stats (used, buffers, cache, free)
      -n, --net
             enable network stats (receive, send)
      -N eth1,total
             include eth1 and total (when using -n/--net)
      -p, --proc
             enable process stats (runnable, uninterruptible, new)
      -r, --io
             enable I/O request stats (read, write requests)
      -s, --swap
             enable swap stats (used, free)
      -S swap1,total
             include swap1 and total (when using -s/--swap)
      -t, --time
             enable time/date output
      -T, --epoch
             enable time counter (seconds since epoch)
      -y, --sys
             enable system stats (interrupts, context switches)
      --fs   enable filesystem stats (open files, inodes)
      --ipc  enable ipc stats (message queue, semaphores, shared memory)
      --lock enable file lock stats (posix, flock, read, write)
      --raw  enable raw stats (raw sockets)
      --socket
             enable socket stats (total, tcp, udp, raw, ip-fragments)
      --tcp  enable tcp stats (listen, established, syn, time_wait, close)
      --udp  enable udp stats (listen, active)
      --unix enable unix stats (datagram, stream, listen, active)
      --vm   enable vm stats (hard pagefaults, soft pagefaults, allocated, free)

Enjoy.