1Demonstrations of memleak. 2 3 4memleak traces and matches memory allocation and deallocation requests, and 5collects call stacks for each allocation. memleak can then print a summary 6of which call stacks performed allocations that weren't subsequently freed. 7For example: 8 9# ./memleak -p $(pidof allocs) 10Attaching to pid 5193, Ctrl+C to quit. 11[11:16:33] Top 2 stacks with outstanding allocations: 12 80 bytes in 5 allocations from stack 13 main+0x6d [allocs] 14 __libc_start_main+0xf0 [libc-2.21.so] 15 16[11:16:34] Top 2 stacks with outstanding allocations: 17 160 bytes in 10 allocations from stack 18 main+0x6d [allocs] 19 __libc_start_main+0xf0 [libc-2.21.so] 20 21 22Each entry printed is a set of allocations that originate from the same call 23stack, and that weren't freed yet. The number of bytes and number of allocs 24are followed by the call stack, top to bottom, of the allocation site. 25 26As time goes on, it becomes apparent that the main function in the allocs 27process is leaking memory, 16 bytes at a time. Fortunately, you don't have to 28inspect each allocation individually -- you get a nice summary of which stack 29is responsible for a large leak. 30 31Occasionally, you do want the individual allocation details. Perhaps the same 32stack is allocating various sizes and you want to confirm which sizes are 33prevalent. Use the -a switch: 34 35# ./memleak -p $(pidof allocs) -a 36Attaching to pid 5193, Ctrl+C to quit. 37[11:16:33] Top 2 stacks with outstanding allocations: 38 addr = 948cd0 size = 16 39 addr = 948d10 size = 16 40 addr = 948d30 size = 16 41 addr = 948cf0 size = 16 42 64 bytes in 4 allocations from stack 43 main+0x6d [allocs] 44 __libc_start_main+0xf0 [libc-2.21.so] 45 46[11:16:34] Top 2 stacks with outstanding allocations: 47 addr = 948d50 size = 16 48 addr = 948cd0 size = 16 49 addr = 948d10 size = 16 50 addr = 948d30 size = 16 51 addr = 948cf0 size = 16 52 addr = 948dd0 size = 16 53 addr = 948d90 size = 16 54 addr = 948db0 size = 16 55 addr = 948d70 size = 16 56 addr = 948df0 size = 16 57 160 bytes in 10 allocations from stack 58 main+0x6d [allocs] 59 __libc_start_main+0xf0 [libc-2.21.so] 60 61 62When using the -p switch, memleak traces the libc allocations of a particular 63process. Without this switch, kernel allocations are traced instead. 64For example: 65 66# ./memleak 67Attaching to kernel allocators, Ctrl+C to quit. 68... 69 248 bytes in 4 allocations from stack 70 bpf_prog_load [kernel] 71 sys_bpf [kernel] 72 73 328 bytes in 1 allocations from stack 74 perf_mmap [kernel] 75 mmap_region [kernel] 76 do_mmap [kernel] 77 vm_mmap_pgoff [kernel] 78 sys_mmap_pgoff [kernel] 79 sys_mmap [kernel] 80 81 464 bytes in 1 allocations from stack 82 traceprobe_command [kernel] 83 traceprobe_probes_write [kernel] 84 probes_write [kernel] 85 __vfs_write [kernel] 86 vfs_write [kernel] 87 sys_write [kernel] 88 entry_SYSCALL_64_fastpath [kernel] 89 90 8192 bytes in 1 allocations from stack 91 alloc_and_copy_ftrace_hash.constprop.59 [kernel] 92 ftrace_set_hash [kernel] 93 ftrace_set_filter_ip [kernel] 94 arm_kprobe [kernel] 95 enable_kprobe [kernel] 96 kprobe_register [kernel] 97 perf_trace_init [kernel] 98 perf_tp_event_init [kernel] 99 100 101Here you can see that arming the kprobe to which our eBPF program is attached 102consumed 8KB of memory. Loading the BPF program also consumed a couple hundred 103bytes (in bpf_prog_load). 104 105memleak stores each allocated block along with its size, timestamp, and the 106stack that allocated it. When the block is deleted, this information is freed 107to reduce the memory overhead. 108 109To avoid false positives, allocations younger than a certain age (500ms by 110default) are not printed. To change this threshold, use the -o switch. 111 112By default, memleak prints its output every 5 seconds. To change this 113interval, pass the interval as a positional parameter to memleak. You can 114also control the number of times the output will be printed before exiting. 115For example: 116 117# ./memleak 1 10 118 119... will print the outstanding allocation statistics every second, for ten 120times, and then exit. 121 122memleak may introduce considerable overhead if your application or kernel is 123allocating and freeing memory at a very high rate. In that case, you can 124control the overhead by sampling every N-th allocation. For example, to sample 125roughly 10% of the allocations and print the outstanding allocations every 5 126seconds, 3 times before quitting: 127 128# ./memleak -p $(pidof allocs) -s 10 5 3 129Attaching to pid 2614, Ctrl+C to quit. 130[11:16:33] Top 2 stacks with outstanding allocations: 131 16 bytes in 1 allocations from stack 132 main+0x6d [allocs] 133 __libc_start_main+0xf0 [libc-2.21.so] 134 135[11:16:38] Top 2 stacks with outstanding allocations: 136 16 bytes in 1 allocations from stack 137 main+0x6d [allocs] 138 __libc_start_main+0xf0 [libc-2.21.so] 139 140[11:16:43] Top 2 stacks with outstanding allocations: 141 32 bytes in 2 allocations from stack 142 main+0x6d [allocs] 143 __libc_start_main+0xf0 [libc-2.21.so] 144 145Note that even though the application leaks 16 bytes of memory every second, 146the report (printed every 5 seconds) doesn't "see" all the allocations because 147of the sampling rate applied. 148 149Profiling in memory part is hard to be accurate because of BPF infrastructure. 150memleak keeps misjudging memory leak on the complicated environment which has 151the action of free in hard/soft irq. 152Add workaround to alleviate misjudgments when free is missing: 153 154# ./memleak --wa-missing-free 155Attaching to kernel allocators, Ctrl+C to quit. 156... 157 248 bytes in 4 allocations from stack 158 bpf_prog_load [kernel] 159 sys_bpf [kernel] 160 161 328 bytes in 1 allocations from stack 162 perf_mmap [kernel] 163 mmap_region [kernel] 164 do_mmap [kernel] 165 vm_mmap_pgoff [kernel] 166 sys_mmap_pgoff [kernel] 167 sys_mmap [kernel] 168 169 170USAGE message: 171 172# ./memleak -h 173usage: memleak.py [-h] [-p PID] [-t] [-a] [-o OLDER] [-c COMMAND] 174 [--combined-only] [--wa-missing-free] [-s SAMPLE_RATE] 175 [-T TOP] [-z MIN_SIZE] [-Z MAX_SIZE] [-O OBJ] 176 [interval] [count] 177 178Trace outstanding memory allocations that weren't freed. 179Supports both user-mode allocations made with libc functions and kernel-mode 180allocations made with kmalloc/kmem_cache_alloc/get_free_pages and corresponding 181memory release functions. 182 183positional arguments: 184 interval interval in seconds to print outstanding allocations 185 count number of times to print the report before exiting 186 187optional arguments: 188 -h, --help show this help message and exit 189 -p PID, --pid PID the PID to trace; if not specified, trace kernel 190 allocs 191 -t, --trace print trace messages for each alloc/free call 192 -a, --show-allocs show allocation addresses and sizes as well as call 193 stacks 194 -o OLDER, --older OLDER 195 prune allocations younger than this age in 196 milliseconds 197 -c COMMAND, --command COMMAND 198 execute and trace the specified command 199 --combined-only show combined allocation statistics only 200 --wa-missing-free Workaround to alleviate misjudgments when free is 201 missing 202 -s SAMPLE_RATE, --sample-rate SAMPLE_RATE 203 sample every N-th allocation to decrease the overhead 204 -T TOP, --top TOP display only this many top allocating stacks (by size) 205 -z MIN_SIZE, --min-size MIN_SIZE 206 capture only allocations larger than this size 207 -Z MAX_SIZE, --max-size MAX_SIZE 208 capture only allocations smaller than this size 209 -O OBJ, --obj OBJ attach to allocator functions in the specified object 210 211EXAMPLES: 212 213./memleak -p $(pidof allocs) 214 Trace allocations and display a summary of "leaked" (outstanding) 215 allocations every 5 seconds 216./memleak -p $(pidof allocs) -t 217 Trace allocations and display each individual allocator function call 218./memleak -ap $(pidof allocs) 10 219 Trace allocations and display allocated addresses, sizes, and stacks 220 every 10 seconds for outstanding allocations 221./memleak -c "./allocs" 222 Run the specified command and trace its allocations 223./memleak 224 Trace allocations in kernel mode and display a summary of outstanding 225 allocations every 5 seconds 226./memleak -o 60000 227 Trace allocations in kernel mode and display a summary of outstanding 228 allocations that are at least one minute (60 seconds) old 229./memleak -s 5 230 Trace roughly every 5th allocation, to reduce overhead 231