1Demonstrations of biotop, the Linux eBPF/bcc version. 2 3 4Short for block device I/O top, biotop summarizes which processes are 5performing disk I/O. It's top for disks. Sample output: 6 7# ./biotop 8Tracing... Output every 1 secs. Hit Ctrl-C to end 9 1008:04:11 loadavg: 1.48 0.87 0.45 1/287 14547 11 12PID COMM D MAJ MIN DISK I/O Kbytes AVGms 1314501 cksum R 202 1 xvda1 361 28832 3.39 146961 dd R 202 1 xvda1 1628 13024 0.59 1513855 dd R 202 1 xvda1 1627 13016 0.59 16326 jbd2/xvda1-8 W 202 1 xvda1 3 168 3.00 171880 supervise W 202 1 xvda1 2 8 6.71 181873 supervise W 202 1 xvda1 2 8 2.51 191871 supervise W 202 1 xvda1 2 8 1.57 201876 supervise W 202 1 xvda1 2 8 1.22 211892 supervise W 202 1 xvda1 2 8 0.62 221878 supervise W 202 1 xvda1 2 8 0.78 231886 supervise W 202 1 xvda1 2 8 1.30 241894 supervise W 202 1 xvda1 2 8 3.46 251869 supervise W 202 1 xvda1 2 8 0.73 261888 supervise W 202 1 xvda1 2 8 1.48 27 28By default the screen refreshes every 1 second, and shows the top 20 disk 29consumers, sorted on total Kbytes. The first line printed is the header, 30which has the time and then the contents of /proc/loadavg. 31 32For the interval summarized by the output above, the "cksum" command performed 33361 disk reads to the "xvda1" device, for a total of 28832 Kbytes, with an 34average I/O time of 3.39 ms. Two "dd" processes were also reading from the 35same disk, which a higher I/O rate and lower latency. While the average I/O 36size is not printed, it can be determined by dividing the Kbytes column by 37the I/O column. 38 39The columns through to Kbytes show the workload applied. The final column, 40AVGms, shows resulting performance. Other bcc tools can be used to get more 41details when needed: biolatency and biosnoop. 42 43Many years ago I created the original "iotop", and later regretted not calling 44it diskiotop or blockiotop, as "io" alone is ambiguous. This time it is biotop. 45 46 47The -C option can be used to prevent the screen from clearing (my preference). 48Here's using it with a 5 second interval: 49 50# ./biotop -C 5 51Tracing... Output every 5 secs. Hit Ctrl-C to end 52 5308:09:44 loadavg: 0.42 0.44 0.39 2/282 22115 54 55PID COMM D MAJ MIN DISK I/O Kbytes AVGms 5622069 dd R 202 1 xvda1 5993 47976 0.33 57326 jbd2/xvda1-8 W 202 1 xvda1 3 168 2.67 581866 svscan R 202 1 xvda1 33 132 1.24 591880 supervise W 202 1 xvda1 10 40 0.56 601873 supervise W 202 1 xvda1 10 40 0.79 611871 supervise W 202 1 xvda1 10 40 0.78 621876 supervise W 202 1 xvda1 10 40 0.68 631892 supervise W 202 1 xvda1 10 40 0.71 641878 supervise W 202 1 xvda1 10 40 0.65 651886 supervise W 202 1 xvda1 10 40 0.78 661894 supervise W 202 1 xvda1 10 40 0.80 671869 supervise W 202 1 xvda1 10 40 0.91 681888 supervise W 202 1 xvda1 10 40 0.63 6922069 bash R 202 1 xvda1 1 16 19.94 709251 kworker/u16:2 W 202 16 xvdb 2 8 0.13 71 7208:09:49 loadavg: 0.47 0.44 0.39 1/282 22231 73 74PID COMM D MAJ MIN DISK I/O Kbytes AVGms 7522069 dd R 202 1 xvda1 13450 107600 0.35 7622199 cksum R 202 1 xvda1 941 45548 4.63 77326 jbd2/xvda1-8 W 202 1 xvda1 3 168 2.93 7824467 kworker/0:2 W 202 16 xvdb 1 64 0.28 791880 supervise W 202 1 xvda1 10 40 0.81 801873 supervise W 202 1 xvda1 10 40 0.81 811871 supervise W 202 1 xvda1 10 40 1.03 821876 supervise W 202 1 xvda1 10 40 0.76 831892 supervise W 202 1 xvda1 10 40 0.74 841878 supervise W 202 1 xvda1 10 40 0.94 851886 supervise W 202 1 xvda1 10 40 0.76 861894 supervise W 202 1 xvda1 10 40 0.69 871869 supervise W 202 1 xvda1 10 40 0.72 881888 supervise W 202 1 xvda1 10 40 1.70 8922199 bash R 202 1 xvda1 2 20 0.35 90482 xfsaild/md0 W 202 16 xvdb 5 13 0.27 91482 xfsaild/md0 W 202 32 xvdc 2 8 0.33 9231331 pickup R 202 1 xvda1 1 4 0.31 93 9408:09:54 loadavg: 0.51 0.45 0.39 2/282 22346 95 96PID COMM D MAJ MIN DISK I/O Kbytes AVGms 9722069 dd R 202 1 xvda1 14689 117512 0.32 98326 jbd2/xvda1-8 W 202 1 xvda1 3 168 2.33 991880 supervise W 202 1 xvda1 10 40 0.65 1001873 supervise W 202 1 xvda1 10 40 1.08 1011871 supervise W 202 1 xvda1 10 40 0.66 1021876 supervise W 202 1 xvda1 10 40 0.79 1031892 supervise W 202 1 xvda1 10 40 0.67 1041878 supervise W 202 1 xvda1 10 40 0.66 1051886 supervise W 202 1 xvda1 10 40 1.02 1061894 supervise W 202 1 xvda1 10 40 0.88 1071869 supervise W 202 1 xvda1 10 40 0.89 1081888 supervise W 202 1 xvda1 10 40 1.25 109 11008:09:59 loadavg: 0.55 0.46 0.40 2/282 22461 111 112PID COMM D MAJ MIN DISK I/O Kbytes AVGms 11322069 dd R 202 1 xvda1 14442 115536 0.33 114326 jbd2/xvda1-8 W 202 1 xvda1 3 168 3.46 1151880 supervise W 202 1 xvda1 10 40 0.87 1161873 supervise W 202 1 xvda1 10 40 0.87 1171871 supervise W 202 1 xvda1 10 40 0.78 1181876 supervise W 202 1 xvda1 10 40 0.86 1191892 supervise W 202 1 xvda1 10 40 0.89 1201878 supervise W 202 1 xvda1 10 40 0.87 1211886 supervise W 202 1 xvda1 10 40 0.86 1221894 supervise W 202 1 xvda1 10 40 1.06 1231869 supervise W 202 1 xvda1 10 40 1.12 1241888 supervise W 202 1 xvda1 10 40 0.98 125 12608:10:04 loadavg: 0.59 0.47 0.40 3/282 22576 127 128PID COMM D MAJ MIN DISK I/O Kbytes AVGms 12922069 dd R 202 1 xvda1 14179 113432 0.34 130326 jbd2/xvda1-8 W 202 1 xvda1 3 168 2.39 1311880 supervise W 202 1 xvda1 10 40 0.81 1321873 supervise W 202 1 xvda1 10 40 1.02 1331871 supervise W 202 1 xvda1 10 40 1.15 1341876 supervise W 202 1 xvda1 10 40 1.10 1351892 supervise W 202 1 xvda1 10 40 0.77 1361878 supervise W 202 1 xvda1 10 40 0.72 1371886 supervise W 202 1 xvda1 10 40 0.81 1381894 supervise W 202 1 xvda1 10 40 0.86 1391869 supervise W 202 1 xvda1 10 40 0.83 1401888 supervise W 202 1 xvda1 10 40 0.79 14124467 kworker/0:2 R 202 32 xvdc 3 12 0.26 1421056 cron R 202 1 xvda1 2 8 0.30 14324467 kworker/0:2 R 202 16 xvdb 1 4 0.23 144 14508:10:09 loadavg: 0.54 0.46 0.40 2/281 22668 146 147PID COMM D MAJ MIN DISK I/O Kbytes AVGms 14822069 dd R 202 1 xvda1 250 2000 0.34 149326 jbd2/xvda1-8 W 202 1 xvda1 3 168 2.40 1501880 supervise W 202 1 xvda1 8 32 0.93 1511873 supervise W 202 1 xvda1 8 32 0.76 1521871 supervise W 202 1 xvda1 8 32 0.60 1531876 supervise W 202 1 xvda1 8 32 0.61 1541892 supervise W 202 1 xvda1 8 32 0.68 1551878 supervise W 202 1 xvda1 8 32 0.90 1561886 supervise W 202 1 xvda1 8 32 0.57 1571894 supervise W 202 1 xvda1 8 32 0.97 1581869 supervise W 202 1 xvda1 8 32 0.69 1591888 supervise W 202 1 xvda1 8 32 0.67 160 161This shows another "dd" command reading from xvda1. On this system, various 162"supervise" processes do 8 disk writes per second, every second (they are 163creating and updating "status" files). 164 165 166USAGE message: 167 168# ./biotop.py -h 169usage: biotop.py [-h] [-C] [-r MAXROWS] [interval] [count] 170 171Block device (disk) I/O by process 172 173positional arguments: 174 interval output interval, in seconds 175 count number of outputs 176 177optional arguments: 178 -h, --help show this help message and exit 179 -C, --noclear don't clear the screen 180 -r MAXROWS, --maxrows MAXROWS 181 maximum rows to print, default 20 182 183examples: 184 ./biotop # block device I/O top, 1 second refresh 185 ./biotop -C # don't clear the screen 186 ./biotop 5 # 5 second summaries 187 ./biotop 5 10 # 5 second summaries, 10 times only 188