1Demonstrations of stacksnoop, the Linux eBPF/bcc version. 2 3 4This program traces the given kernel function and prints the kernel stack trace 5for every call. This tool is useful for studying low frequency kernel functions, 6to see how they were invoked. For example, tracing the submit_bio() call: 7 8# ./stacksnoop submit_bio 9TIME(s) SYSCALL 103592.838736000 submit_bio 11 submit_bio 12 submit_bh 13 jbd2_journal_commit_transaction 14 kjournald2 15 kthread 16 ret_from_fork 17 18This shows that submit_bio() was called by submit_bh(), which was called 19by jbd2_journal_commit_transaction(), and so on. 20 21For high frequency functions, see stackcount, which summarizes in-kernel for 22efficiency. If you don't know if your function is low or high frequency, try 23funccount. 24 25 26The -v option includes more fields, including the on-CPU process (COMM and PID): 27 28# ./stacksnoop -v submit_bio 29TIME(s) COMM PID CPU SYSCALL 303734.855027000 jbd2/dm-0-8 313 0 submit_bio 31 submit_bio 32 submit_bh 33 jbd2_journal_commit_transaction 34 kjournald2 35 kthread 36 ret_from_fork 37 38This identifies the application issuing the sync syscall: the jbd2 process 39(COMM column). 40 41 42Here's another example, showing the path to second_overflow() and on-CPU 43process: 44 45# ./stacksnoop -v second_overflow 46TIME(s) COMM PID CPU SYSCALL 473837.526433000 <idle> 0 1 second_overflow 48 second_overflow 49 tick_do_update_jiffies64 50 tick_irq_enter 51 irq_enter 52 smp_apic_timer_interrupt 53 apic_timer_interrupt 54 default_idle 55 arch_cpu_idle 56 default_idle_call 57 cpu_startup_entry 58 start_secondary 59 603838.526953000 <idle> 0 1 second_overflow 61 second_overflow 62 tick_do_update_jiffies64 63 tick_irq_enter 64 irq_enter 65 smp_apic_timer_interrupt 66 apic_timer_interrupt 67 default_idle 68 arch_cpu_idle 69 default_idle_call 70 cpu_startup_entry 71 start_secondary 72 73This fires every second (see TIME(s)), and is from tick_do_update_jiffies64(). 74 75 76USAGE message: 77 78# ./stacksnoop -h 79usage: stacksnoop [-h] [-p PID] [-s] [-v] function 80 81Trace and print kernel stack traces for a kernel function 82 83positional arguments: 84 function kernel function name 85 86optional arguments: 87 -h, --help show this help message and exit 88 -p PID, --pid PID trace this PID only 89 -s, --offset show address offsets 90 -v, --verbose print more fields 91 92examples: 93 ./stacksnoop ext4_sync_fs # print kernel stack traces for ext4_sync_fs 94 ./stacksnoop -s ext4_sync_fs # ... also show symbol offsets 95 ./stacksnoop -v ext4_sync_fs # ... show extra columns 96 ./stacksnoop -p 185 ext4_sync_fs # ... only when PID 185 is on-CPU 97