memleak - Print a summary of outstanding allocations and their call stacks to detect memory leaks. Uses Linux eBPF/bcc. memleak [-h] [-p PID] [-t] [-a] [-o OLDER] [-c COMMAND] [--combined-only] [-s SAMPLE_RATE] [-T TOP] [-z MIN_SIZE] [-Z MAX_SIZE] [-O OBJ] [INTERVAL] [COUNT] memleak traces and matches memory allocation and deallocation requests, and collects call stacks for each allocation. memleak can then print a summary of which call stacks performed allocations that weren't subsequently freed. When tracing a specific process, memleak instruments a list of allocation functions from libc, specifically: malloc, calloc, realloc, posix_memalign, valloc, memalign, pvalloc, aligned_alloc, and free. When tracing all processes, memleak instruments kmalloc/kfree, kmem_cache_alloc/kmem_cache_free, and also page allocations made by get_free_pages/free_pages. memleak may introduce significant overhead when tracing processes that allocate and free many blocks very quickly. See the OVERHEAD section below. This tool only works on Linux 4.6+. Stack traces are obtained using the new BPF_STACK_TRACE` APIs. For kernels older than 4.6, see the version under tools/old. Kernel memory allocations are intercepted through tracepoints, which are available on Linux 4.7+. CONFIG_BPF and bcc.

-h Print usage message.

-p PID Trace this process ID only (filtered in-kernel). This traces libc allocator.

-t Print a trace of all allocation and free requests and results.

-a Print a list of allocations that weren't freed (and their sizes) in addition to their call stacks.

-o OLDER Print only allocations older than OLDER milliseconds. Useful to remove false positives. The default value is 500 milliseconds.

-c COMMAND Run the specified command and trace its allocations only. This traces libc allocator.

--combined-only Use statistics precalculated in kernel space. Amount of data to be pulled from kernel significantly decreases, at the cost of losing capabilities of time-based false positives filtering (-o).

-s SAMPLE_RATE Record roughly every SAMPLE_RATE-th allocation to reduce overhead.

-t TOP Print only the top TOP stacks (sorted by size). The default value is 10.

-z MIN_SIZE Capture only allocations that are larger than or equal to MIN_SIZE bytes.

-Z MAX_SIZE Capture only allocations that are smaller than or equal to MAX_SIZE bytes.

-O OBJ Attach to allocation functions in specified object instead of resolving libc. Ignored when kernel allocations are profiled.

INTERVAL Print a summary of outstanding allocations and their call stacks every INTERVAL seconds. The default interval is 5 seconds.

COUNT Print the outstanding allocations summary COUNT times and then exit.

Print outstanding kernel allocation stacks every 3 seconds: # memleak 3

Print user outstanding allocation stacks and allocation details for the process 1005: # memleak -p 1005 -a

Sample roughly every 5th allocation (~20%) of the call stacks and print the top 5 stacks 10 times before quitting. # memleak -s 5 --top=5 10

Run ./allocs and print outstanding allocation stacks for that process: # memleak -c "./allocs"

Capture only allocations between 16 and 32 bytes in size: # memleak -z 16 -Z 32

memleak can have significant overhead if the target process or kernel performs allocations at a very high rate. Pathological cases may exhibit up to 100x degradation in running time. Most of the time, however, memleak shouldn't cause a significant slowdown. You can use the -s switch to reduce the overhead further by capturing only every N-th allocation. The -z and -Z switches can also reduce overhead by capturing only allocations of specific sizes. Additionally, option --combined-only saves processing time by reusing already calculated allocation statistics from kernel. It's faster, but lacks information about particular allocations. To determine the rate at which your application is calling malloc/free, or the rate at which your kernel is calling kmalloc/kfree, place a probe with perf and collect statistics. For example, to determine how many calls to __kmalloc are placed in a typical period of 10 seconds: # perf probe '__kmalloc' # perf stat -a -e 'probe:__kmalloc' -- sleep 10 This is from bcc. https://github.com/iovisor/bcc

Also look in the bcc distribution for a companion _examples.txt file containing example usage, output, and commentary for this tool.

Linux Unstable - in development. Sasha Goldshtein