1 2Introduction 3============ 4 5This document describes how to use the dynamic debug (dyndbg) feature. 6 7Dynamic debug is designed to allow you to dynamically enable/disable 8kernel code to obtain additional kernel information. Currently, if 9CONFIG_DYNAMIC_DEBUG is set, then all pr_debug()/dev_dbg() and 10print_hex_dump_debug()/print_hex_dump_bytes() calls can be dynamically 11enabled per-callsite. 12 13If CONFIG_DYNAMIC_DEBUG is not set, print_hex_dump_debug() is just 14shortcut for print_hex_dump(KERN_DEBUG). 15 16For print_hex_dump_debug()/print_hex_dump_bytes(), format string is 17its 'prefix_str' argument, if it is constant string; or "hexdump" 18in case 'prefix_str' is build dynamically. 19 20Dynamic debug has even more useful features: 21 22 * Simple query language allows turning on and off debugging 23 statements by matching any combination of 0 or 1 of: 24 25 - source filename 26 - function name 27 - line number (including ranges of line numbers) 28 - module name 29 - format string 30 31 * Provides a debugfs control file: <debugfs>/dynamic_debug/control 32 which can be read to display the complete list of known debug 33 statements, to help guide you 34 35Controlling dynamic debug Behaviour 36=================================== 37 38The behaviour of pr_debug()/dev_dbg()s are controlled via writing to a 39control file in the 'debugfs' filesystem. Thus, you must first mount 40the debugfs filesystem, in order to make use of this feature. 41Subsequently, we refer to the control file as: 42<debugfs>/dynamic_debug/control. For example, if you want to enable 43printing from source file 'svcsock.c', line 1603 you simply do: 44 45nullarbor:~ # echo 'file svcsock.c line 1603 +p' > 46 <debugfs>/dynamic_debug/control 47 48If you make a mistake with the syntax, the write will fail thus: 49 50nullarbor:~ # echo 'file svcsock.c wtf 1 +p' > 51 <debugfs>/dynamic_debug/control 52-bash: echo: write error: Invalid argument 53 54Viewing Dynamic Debug Behaviour 55=========================== 56 57You can view the currently configured behaviour of all the debug 58statements via: 59 60nullarbor:~ # cat <debugfs>/dynamic_debug/control 61# filename:lineno [module]function flags format 62/usr/src/packages/BUILD/sgi-enhancednfs-1.4/default/net/sunrpc/svc_rdma.c:323 [svcxprt_rdma]svc_rdma_cleanup =_ "SVCRDMA Module Removed, deregister RPC RDMA transport\012" 63/usr/src/packages/BUILD/sgi-enhancednfs-1.4/default/net/sunrpc/svc_rdma.c:341 [svcxprt_rdma]svc_rdma_init =_ "\011max_inline : %d\012" 64/usr/src/packages/BUILD/sgi-enhancednfs-1.4/default/net/sunrpc/svc_rdma.c:340 [svcxprt_rdma]svc_rdma_init =_ "\011sq_depth : %d\012" 65/usr/src/packages/BUILD/sgi-enhancednfs-1.4/default/net/sunrpc/svc_rdma.c:338 [svcxprt_rdma]svc_rdma_init =_ "\011max_requests : %d\012" 66... 67 68 69You can also apply standard Unix text manipulation filters to this 70data, e.g. 71 72nullarbor:~ # grep -i rdma <debugfs>/dynamic_debug/control | wc -l 7362 74 75nullarbor:~ # grep -i tcp <debugfs>/dynamic_debug/control | wc -l 7642 77 78The third column shows the currently enabled flags for each debug 79statement callsite (see below for definitions of the flags). The 80default value, with no flags enabled, is "=_". So you can view all 81the debug statement callsites with any non-default flags: 82 83nullarbor:~ # awk '$3 != "=_"' <debugfs>/dynamic_debug/control 84# filename:lineno [module]function flags format 85/usr/src/packages/BUILD/sgi-enhancednfs-1.4/default/net/sunrpc/svcsock.c:1603 [sunrpc]svc_send p "svc_process: st_sendto returned %d\012" 86 87 88Command Language Reference 89========================== 90 91At the lexical level, a command comprises a sequence of words separated 92by spaces or tabs. So these are all equivalent: 93 94nullarbor:~ # echo -c 'file svcsock.c line 1603 +p' > 95 <debugfs>/dynamic_debug/control 96nullarbor:~ # echo -c ' file svcsock.c line 1603 +p ' > 97 <debugfs>/dynamic_debug/control 98nullarbor:~ # echo -n 'file svcsock.c line 1603 +p' > 99 <debugfs>/dynamic_debug/control 100 101Command submissions are bounded by a write() system call. 102Multiple commands can be written together, separated by ';' or '\n'. 103 104 ~# echo "func pnpacpi_get_resources +p; func pnp_assign_mem +p" \ 105 > <debugfs>/dynamic_debug/control 106 107If your query set is big, you can batch them too: 108 109 ~# cat query-batch-file > <debugfs>/dynamic_debug/control 110 111At the syntactical level, a command comprises a sequence of match 112specifications, followed by a flags change specification. 113 114command ::= match-spec* flags-spec 115 116The match-spec's are used to choose a subset of the known pr_debug() 117callsites to which to apply the flags-spec. Think of them as a query 118with implicit ANDs between each pair. Note that an empty list of 119match-specs will select all debug statement callsites. 120 121A match specification comprises a keyword, which controls the 122attribute of the callsite to be compared, and a value to compare 123against. Possible keywords are: 124 125match-spec ::= 'func' string | 126 'file' string | 127 'module' string | 128 'format' string | 129 'line' line-range 130 131line-range ::= lineno | 132 '-'lineno | 133 lineno'-' | 134 lineno'-'lineno 135// Note: line-range cannot contain space, e.g. 136// "1-30" is valid range but "1 - 30" is not. 137 138lineno ::= unsigned-int 139 140The meanings of each keyword are: 141 142func 143 The given string is compared against the function name 144 of each callsite. Example: 145 146 func svc_tcp_accept 147 148file 149 The given string is compared against either the full pathname, the 150 src-root relative pathname, or the basename of the source file of 151 each callsite. Examples: 152 153 file svcsock.c 154 file kernel/freezer.c 155 file /usr/src/packages/BUILD/sgi-enhancednfs-1.4/default/net/sunrpc/svcsock.c 156 157module 158 The given string is compared against the module name 159 of each callsite. The module name is the string as 160 seen in "lsmod", i.e. without the directory or the .ko 161 suffix and with '-' changed to '_'. Examples: 162 163 module sunrpc 164 module nfsd 165 166format 167 The given string is searched for in the dynamic debug format 168 string. Note that the string does not need to match the 169 entire format, only some part. Whitespace and other 170 special characters can be escaped using C octal character 171 escape \ooo notation, e.g. the space character is \040. 172 Alternatively, the string can be enclosed in double quote 173 characters (") or single quote characters ('). 174 Examples: 175 176 format svcrdma: // many of the NFS/RDMA server pr_debugs 177 format readahead // some pr_debugs in the readahead cache 178 format nfsd:\040SETATTR // one way to match a format with whitespace 179 format "nfsd: SETATTR" // a neater way to match a format with whitespace 180 format 'nfsd: SETATTR' // yet another way to match a format with whitespace 181 182line 183 The given line number or range of line numbers is compared 184 against the line number of each pr_debug() callsite. A single 185 line number matches the callsite line number exactly. A 186 range of line numbers matches any callsite between the first 187 and last line number inclusive. An empty first number means 188 the first line in the file, an empty line number means the 189 last number in the file. Examples: 190 191 line 1603 // exactly line 1603 192 line 1600-1605 // the six lines from line 1600 to line 1605 193 line -1605 // the 1605 lines from line 1 to line 1605 194 line 1600- // all lines from line 1600 to the end of the file 195 196The flags specification comprises a change operation followed 197by one or more flag characters. The change operation is one 198of the characters: 199 200 - remove the given flags 201 + add the given flags 202 = set the flags to the given flags 203 204The flags are: 205 206 p enables the pr_debug() callsite. 207 f Include the function name in the printed message 208 l Include line number in the printed message 209 m Include module name in the printed message 210 t Include thread ID in messages not generated from interrupt context 211 _ No flags are set. (Or'd with others on input) 212 213For print_hex_dump_debug() and print_hex_dump_bytes(), only 'p' flag 214have meaning, other flags ignored. 215 216For display, the flags are preceded by '=' 217(mnemonic: what the flags are currently equal to). 218 219Note the regexp ^[-+=][flmpt_]+$ matches a flags specification. 220To clear all flags at once, use "=_" or "-flmpt". 221 222 223Debug messages during Boot Process 224================================== 225 226To activate debug messages for core code and built-in modules during 227the boot process, even before userspace and debugfs exists, use 228dyndbg="QUERY", module.dyndbg="QUERY", or ddebug_query="QUERY" 229(ddebug_query is obsoleted by dyndbg, and deprecated). QUERY follows 230the syntax described above, but must not exceed 1023 characters. Your 231bootloader may impose lower limits. 232 233These dyndbg params are processed just after the ddebug tables are 234processed, as part of the arch_initcall. Thus you can enable debug 235messages in all code run after this arch_initcall via this boot 236parameter. 237 238On an x86 system for example ACPI enablement is a subsys_initcall and 239 dyndbg="file ec.c +p" 240will show early Embedded Controller transactions during ACPI setup if 241your machine (typically a laptop) has an Embedded Controller. 242PCI (or other devices) initialization also is a hot candidate for using 243this boot parameter for debugging purposes. 244 245If foo module is not built-in, foo.dyndbg will still be processed at 246boot time, without effect, but will be reprocessed when module is 247loaded later. dyndbg_query= and bare dyndbg= are only processed at 248boot. 249 250 251Debug Messages at Module Initialization Time 252============================================ 253 254When "modprobe foo" is called, modprobe scans /proc/cmdline for 255foo.params, strips "foo.", and passes them to the kernel along with 256params given in modprobe args or /etc/modprob.d/*.conf files, 257in the following order: 258 2591. # parameters given via /etc/modprobe.d/*.conf 260 options foo dyndbg=+pt 261 options foo dyndbg # defaults to +p 262 2632. # foo.dyndbg as given in boot args, "foo." is stripped and passed 264 foo.dyndbg=" func bar +p; func buz +mp" 265 2663. # args to modprobe 267 modprobe foo dyndbg==pmf # override previous settings 268 269These dyndbg queries are applied in order, with last having final say. 270This allows boot args to override or modify those from /etc/modprobe.d 271(sensible, since 1 is system wide, 2 is kernel or boot specific), and 272modprobe args to override both. 273 274In the foo.dyndbg="QUERY" form, the query must exclude "module foo". 275"foo" is extracted from the param-name, and applied to each query in 276"QUERY", and only 1 match-spec of each type is allowed. 277 278The dyndbg option is a "fake" module parameter, which means: 279 280- modules do not need to define it explicitly 281- every module gets it tacitly, whether they use pr_debug or not 282- it doesnt appear in /sys/module/$module/parameters/ 283 To see it, grep the control file, or inspect /proc/cmdline. 284 285For CONFIG_DYNAMIC_DEBUG kernels, any settings given at boot-time (or 286enabled by -DDEBUG flag during compilation) can be disabled later via 287the sysfs interface if the debug messages are no longer needed: 288 289 echo "module module_name -p" > <debugfs>/dynamic_debug/control 290 291Examples 292======== 293 294// enable the message at line 1603 of file svcsock.c 295nullarbor:~ # echo -n 'file svcsock.c line 1603 +p' > 296 <debugfs>/dynamic_debug/control 297 298// enable all the messages in file svcsock.c 299nullarbor:~ # echo -n 'file svcsock.c +p' > 300 <debugfs>/dynamic_debug/control 301 302// enable all the messages in the NFS server module 303nullarbor:~ # echo -n 'module nfsd +p' > 304 <debugfs>/dynamic_debug/control 305 306// enable all 12 messages in the function svc_process() 307nullarbor:~ # echo -n 'func svc_process +p' > 308 <debugfs>/dynamic_debug/control 309 310// disable all 12 messages in the function svc_process() 311nullarbor:~ # echo -n 'func svc_process -p' > 312 <debugfs>/dynamic_debug/control 313 314// enable messages for NFS calls READ, READLINK, READDIR and READDIR+. 315nullarbor:~ # echo -n 'format "nfsd: READ" +p' > 316 <debugfs>/dynamic_debug/control 317 318// enable all messages 319nullarbor:~ # echo -n '+p' > <debugfs>/dynamic_debug/control 320 321// add module, function to all enabled messages 322nullarbor:~ # echo -n '+mf' > <debugfs>/dynamic_debug/control 323 324// boot-args example, with newlines and comments for readability 325Kernel command line: ... 326 // see whats going on in dyndbg=value processing 327 dynamic_debug.verbose=1 328 // enable pr_debugs in 2 builtins, #cmt is stripped 329 dyndbg="module params +p #cmt ; module sys +p" 330 // enable pr_debugs in 2 functions in a module loaded later 331 pc87360.dyndbg="func pc87360_init_device +p; func pc87360_find +p" 332