1Demonstrations of biosnoop, the Linux eBPF/bcc version. 2 3 4biosnoop traces block device I/O (disk I/O), and prints a line of output 5per I/O. Example: 6 7# ./biosnoop 8TIME(s) COMM PID DISK T SECTOR BYTES LAT(ms) 90.000004001 supervise 1950 xvda1 W 13092560 4096 0.74 100.000178002 supervise 1950 xvda1 W 13092432 4096 0.61 110.001469001 supervise 1956 xvda1 W 13092440 4096 1.24 120.001588002 supervise 1956 xvda1 W 13115128 4096 1.09 131.022346001 supervise 1950 xvda1 W 13115272 4096 0.98 141.022568002 supervise 1950 xvda1 W 13188496 4096 0.93 151.023534000 supervise 1956 xvda1 W 13188520 4096 0.79 161.023585003 supervise 1956 xvda1 W 13189512 4096 0.60 172.003920000 xfsaild/md0 456 xvdc W 62901512 8192 0.23 182.003931001 xfsaild/md0 456 xvdb W 62901513 512 0.25 192.004034001 xfsaild/md0 456 xvdb W 62901520 8192 0.35 202.004042000 xfsaild/md0 456 xvdb W 63542016 4096 0.36 212.004204001 kworker/0:3 26040 xvdb W 41950344 65536 0.34 222.044352002 supervise 1950 xvda1 W 13192672 4096 0.65 232.044574000 supervise 1950 xvda1 W 13189072 4096 0.58 24 25This includes the PID and comm (process name) that were on-CPU at the time of 26issue (which usually means the process responsible). 27 28The latency of the disk I/O, measured from the issue to the device to its 29completion, is included as the last column. 30 31This example output is from what should be an idle system, however, the 32following is visible in iostat: 33 34$ iostat -x 1 35[...] 36avg-cpu: %user %nice %system %iowait %steal %idle 37 0.12 0.00 0.12 0.00 0.00 99.75 38 39Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s await svctm %util 40xvda 0.00 0.00 0.00 4.00 0.00 16.00 0.00 0.00 0.00 41xvdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 42xvdc 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 43md0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 44 45There are 4 write IOPS. 46 47The output of biosnoop identifies the reason: multiple supervise processes are 48issuing writes to the xvda1 disk. I can now drill down on supervise using other 49tools to understand its file system workload. 50