1Demonstrations of tplist. 2 3 4tplist displays kernel tracepoints and USDT probes, including their 5format. It can be used to discover probe points for use with the trace 6and argdist tools. Kernel tracepoints are scattered around the kernel 7and provide valuable static tracing on block and network I/O, scheduling, 8power events, and many other subjects. USDT probes are placed in libraries 9(such as libc) and executables (such as node) and provide static tracing 10information that can (optionally) be turned on and off at runtime. 11 12For example, suppose you want to discover which USDT probes a particular 13executable contains. Just run tplist on that executable (or library): 14 15$ tplist -l basic_usdt 16/home/vagrant/basic_usdt basic_usdt:start_main 17/home/vagrant/basic_usdt basic_usdt:loop_iter 18/home/vagrant/basic_usdt basic_usdt:end_main 19 20The loop_iter probe sounds interesting. How many arguments are available? 21 22$ tplist '*loop_iter' -l basic_usdt -v 23basic_usdt:loop_iter [sema 0x601036] 24 2 location(s) 25 2 argument(s) 26 27This output indicates that the loop_iter probe is used in two locations 28in the basic_usdt executable, and that it has two arguments. Fortunately, 29the argdist and trace tools understand the probe format and can print out 30the arguments automatically -- you can refer to them as arg1, arg2, and 31so on. 32 33Try to explore with some common libraries on your system and see if they 34contain UDST probes. Here are two examples you might find interesting: 35 36$ tplist -l pthread # list probes in libpthread 37/lib64/libpthread.so.0 libpthread:pthread_start 38/lib64/libpthread.so.0 libpthread:pthread_create 39/lib64/libpthread.so.0 libpthread:pthread_join 40/lib64/libpthread.so.0 libpthread:pthread_join_ret 41/lib64/libpthread.so.0 libpthread:mutex_init 42... more output truncated 43 44$ tplist -l c # list probes in libc 45/lib64/libc.so.6 libc:setjmp 46/lib64/libc.so.6 libc:longjmp 47/lib64/libc.so.6 libc:longjmp_target 48/lib64/libc.so.6 libc:memory_arena_reuse_free_list 49/lib64/libc.so.6 libc:memory_heap_new 50... more output truncated 51 52tplist also understands kernel tracepoints, and can list their format 53as well. For example, let's look for all block I/O-related tracepoints: 54 55# tplist 'block*' 56block:block_touch_buffer 57block:block_dirty_buffer 58block:block_rq_abort 59block:block_rq_requeue 60block:block_rq_complete 61block:block_rq_insert 62block:block_rq_issue 63block:block_bio_bounce 64block:block_bio_complete 65block:block_bio_backmerge 66block:block_bio_frontmerge 67block:block_bio_queue 68block:block_getrq 69block:block_sleeprq 70block:block_plug 71block:block_unplug 72block:block_split 73block:block_bio_remap 74block:block_rq_remap 75 76The block:block_rq_complete tracepoints sounds interesting. Let's print 77its format to see what we can trace with argdist and trace: 78 79$ tplist -v block:block_rq_complete 80block:block_rq_complete 81 dev_t dev; 82 sector_t sector; 83 unsigned int nr_sector; 84 int errors; 85 char rwbs[8]; 86 87The dev, sector, nr_sector, etc. variables can now all be used in probes 88you specify with argdist or trace. 89 90 91For debugging USDT probes, it is sometimes useful to see the exact locations 92and arguments of the probes, including the registers or global variables from 93which their values are coming from. In super-verbose mode, tplist will print 94this information (note the -vv): 95 96$ tplist -vv -l c *alloc* 97libc:memory_malloc_retry [sema 0x0] 98 location #0 /lib64/libc.so.6 0x835c0 99 argument #0 8 unsigned bytes @ bp 100 location #1 /lib64/libc.so.6 0x83778 101 argument #0 8 unsigned bytes @ bp 102 location #2 /lib64/libc.so.6 0x85a50 103 argument #0 8 unsigned bytes @ bp 104libc:memory_realloc_retry [sema 0x0] 105 location #0 /lib64/libc.so.6 0x84b90 106 argument #0 8 unsigned bytes @ r13 107 argument #1 8 unsigned bytes @ bp 108 location #1 /lib64/libc.so.6 0x85cf0 109 argument #0 8 unsigned bytes @ r13 110 argument #1 8 unsigned bytes @ bp 111libc:memory_calloc_retry [sema 0x0] 112 location #0 /lib64/libc.so.6 0x850f0 113 argument #0 8 unsigned bytes @ bp 114 115 116USAGE message: 117 118$ tplist -h 119usage: tplist.py [-h] [-p PID] [-l LIB] [-v] [filter] 120 121Display kernel tracepoints or USDT probes and their formats. 122 123positional arguments: 124 filter A filter that specifies which probes/tracepoints to print 125 126optional arguments: 127 -h, --help show this help message and exit 128 -p PID, --pid PID List USDT probes in the specified process 129 -l LIB, --lib LIB List USDT probes in the specified library or executable 130 -v Increase verbosity level (print variables, arguments, etc.) 131 132