1=========================================== 2Fault injection capabilities infrastructure 3=========================================== 4 5See also drivers/md/md-faulty.c and "every_nth" module option for scsi_debug. 6 7 8Available fault injection capabilities 9-------------------------------------- 10 11- failslab 12 13 injects slab allocation failures. (kmalloc(), kmem_cache_alloc(), ...) 14 15- fail_page_alloc 16 17 injects page allocation failures. (alloc_pages(), get_free_pages(), ...) 18 19- fail_usercopy 20 21 injects failures in user memory access functions. (copy_from_user(), get_user(), ...) 22 23- fail_futex 24 25 injects futex deadlock and uaddr fault errors. 26 27- fail_sunrpc 28 29 injects kernel RPC client and server failures. 30 31- fail_make_request 32 33 injects disk IO errors on devices permitted by setting 34 /sys/block/<device>/make-it-fail or 35 /sys/block/<device>/<partition>/make-it-fail. (submit_bio_noacct()) 36 37- fail_mmc_request 38 39 injects MMC data errors on devices permitted by setting 40 debugfs entries under /sys/kernel/debug/mmc0/fail_mmc_request 41 42- fail_function 43 44 injects error return on specific functions, which are marked by 45 ALLOW_ERROR_INJECTION() macro, by setting debugfs entries 46 under /sys/kernel/debug/fail_function. No boot option supported. 47 48- NVMe fault injection 49 50 inject NVMe status code and retry flag on devices permitted by setting 51 debugfs entries under /sys/kernel/debug/nvme*/fault_inject. The default 52 status code is NVME_SC_INVALID_OPCODE with no retry. The status code and 53 retry flag can be set via the debugfs. 54 55 56Configure fault-injection capabilities behavior 57----------------------------------------------- 58 59debugfs entries 60^^^^^^^^^^^^^^^ 61 62fault-inject-debugfs kernel module provides some debugfs entries for runtime 63configuration of fault-injection capabilities. 64 65- /sys/kernel/debug/fail*/probability: 66 67 likelihood of failure injection, in percent. 68 69 Format: <percent> 70 71 Note that one-failure-per-hundred is a very high error rate 72 for some testcases. Consider setting probability=100 and configure 73 /sys/kernel/debug/fail*/interval for such testcases. 74 75- /sys/kernel/debug/fail*/interval: 76 77 specifies the interval between failures, for calls to 78 should_fail() that pass all the other tests. 79 80 Note that if you enable this, by setting interval>1, you will 81 probably want to set probability=100. 82 83- /sys/kernel/debug/fail*/times: 84 85 specifies how many times failures may happen at most. A value of -1 86 means "no limit". 87 88- /sys/kernel/debug/fail*/space: 89 90 specifies an initial resource "budget", decremented by "size" 91 on each call to should_fail(,size). Failure injection is 92 suppressed until "space" reaches zero. 93 94- /sys/kernel/debug/fail*/verbose 95 96 Format: { 0 | 1 | 2 } 97 98 specifies the verbosity of the messages when failure is 99 injected. '0' means no messages; '1' will print only a single 100 log line per failure; '2' will print a call trace too -- useful 101 to debug the problems revealed by fault injection. 102 103- /sys/kernel/debug/fail*/task-filter: 104 105 Format: { 'Y' | 'N' } 106 107 A value of 'N' disables filtering by process (default). 108 Any positive value limits failures to only processes indicated by 109 /proc/<pid>/make-it-fail==1. 110 111- /sys/kernel/debug/fail*/require-start, 112 /sys/kernel/debug/fail*/require-end, 113 /sys/kernel/debug/fail*/reject-start, 114 /sys/kernel/debug/fail*/reject-end: 115 116 specifies the range of virtual addresses tested during 117 stacktrace walking. Failure is injected only if some caller 118 in the walked stacktrace lies within the required range, and 119 none lies within the rejected range. 120 Default required range is [0,ULONG_MAX) (whole of virtual address space). 121 Default rejected range is [0,0). 122 123- /sys/kernel/debug/fail*/stacktrace-depth: 124 125 specifies the maximum stacktrace depth walked during search 126 for a caller within [require-start,require-end) OR 127 [reject-start,reject-end). 128 129- /sys/kernel/debug/fail_page_alloc/ignore-gfp-highmem: 130 131 Format: { 'Y' | 'N' } 132 133 default is 'N', setting it to 'Y' won't inject failures into 134 highmem/user allocations. 135 136- /sys/kernel/debug/failslab/ignore-gfp-wait: 137- /sys/kernel/debug/fail_page_alloc/ignore-gfp-wait: 138 139 Format: { 'Y' | 'N' } 140 141 default is 'N', setting it to 'Y' will inject failures 142 only into non-sleep allocations (GFP_ATOMIC allocations). 143 144- /sys/kernel/debug/fail_page_alloc/min-order: 145 146 specifies the minimum page allocation order to be injected 147 failures. 148 149- /sys/kernel/debug/fail_futex/ignore-private: 150 151 Format: { 'Y' | 'N' } 152 153 default is 'N', setting it to 'Y' will disable failure injections 154 when dealing with private (address space) futexes. 155 156- /sys/kernel/debug/fail_sunrpc/ignore-client-disconnect: 157 158 Format: { 'Y' | 'N' } 159 160 default is 'N', setting it to 'Y' will disable disconnect 161 injection on the RPC client. 162 163- /sys/kernel/debug/fail_sunrpc/ignore-server-disconnect: 164 165 Format: { 'Y' | 'N' } 166 167 default is 'N', setting it to 'Y' will disable disconnect 168 injection on the RPC server. 169 170- /sys/kernel/debug/fail_function/inject: 171 172 Format: { 'function-name' | '!function-name' | '' } 173 174 specifies the target function of error injection by name. 175 If the function name leads '!' prefix, given function is 176 removed from injection list. If nothing specified ('') 177 injection list is cleared. 178 179- /sys/kernel/debug/fail_function/injectable: 180 181 (read only) shows error injectable functions and what type of 182 error values can be specified. The error type will be one of 183 below; 184 - NULL: retval must be 0. 185 - ERRNO: retval must be -1 to -MAX_ERRNO (-4096). 186 - ERR_NULL: retval must be 0 or -1 to -MAX_ERRNO (-4096). 187 188- /sys/kernel/debug/fail_function/<function-name>/retval: 189 190 specifies the "error" return value to inject to the given function. 191 This will be created when the user specifies a new injection entry. 192 Note that this file only accepts unsigned values. So, if you want to 193 use a negative errno, you better use 'printf' instead of 'echo', e.g.: 194 $ printf %#x -12 > retval 195 196Boot option 197^^^^^^^^^^^ 198 199In order to inject faults while debugfs is not available (early boot time), 200use the boot option:: 201 202 failslab= 203 fail_page_alloc= 204 fail_usercopy= 205 fail_make_request= 206 fail_futex= 207 mmc_core.fail_request=<interval>,<probability>,<space>,<times> 208 209proc entries 210^^^^^^^^^^^^ 211 212- /proc/<pid>/fail-nth, 213 /proc/self/task/<tid>/fail-nth: 214 215 Write to this file of integer N makes N-th call in the task fail. 216 Read from this file returns a integer value. A value of '0' indicates 217 that the fault setup with a previous write to this file was injected. 218 A positive integer N indicates that the fault wasn't yet injected. 219 Note that this file enables all types of faults (slab, futex, etc). 220 This setting takes precedence over all other generic debugfs settings 221 like probability, interval, times, etc. But per-capability settings 222 (e.g. fail_futex/ignore-private) take precedence over it. 223 224 This feature is intended for systematic testing of faults in a single 225 system call. See an example below. 226 227How to add new fault injection capability 228----------------------------------------- 229 230- #include <linux/fault-inject.h> 231 232- define the fault attributes 233 234 DECLARE_FAULT_ATTR(name); 235 236 Please see the definition of struct fault_attr in fault-inject.h 237 for details. 238 239- provide a way to configure fault attributes 240 241- boot option 242 243 If you need to enable the fault injection capability from boot time, you can 244 provide boot option to configure it. There is a helper function for it: 245 246 setup_fault_attr(attr, str); 247 248- debugfs entries 249 250 failslab, fail_page_alloc, fail_usercopy, and fail_make_request use this way. 251 Helper functions: 252 253 fault_create_debugfs_attr(name, parent, attr); 254 255- module parameters 256 257 If the scope of the fault injection capability is limited to a 258 single kernel module, it is better to provide module parameters to 259 configure the fault attributes. 260 261- add a hook to insert failures 262 263 Upon should_fail() returning true, client code should inject a failure: 264 265 should_fail(attr, size); 266 267Application Examples 268-------------------- 269 270- Inject slab allocation failures into module init/exit code:: 271 272 #!/bin/bash 273 274 FAILTYPE=failslab 275 echo Y > /sys/kernel/debug/$FAILTYPE/task-filter 276 echo 10 > /sys/kernel/debug/$FAILTYPE/probability 277 echo 100 > /sys/kernel/debug/$FAILTYPE/interval 278 echo -1 > /sys/kernel/debug/$FAILTYPE/times 279 echo 0 > /sys/kernel/debug/$FAILTYPE/space 280 echo 2 > /sys/kernel/debug/$FAILTYPE/verbose 281 echo 1 > /sys/kernel/debug/$FAILTYPE/ignore-gfp-wait 282 283 faulty_system() 284 { 285 bash -c "echo 1 > /proc/self/make-it-fail && exec $*" 286 } 287 288 if [ $# -eq 0 ] 289 then 290 echo "Usage: $0 modulename [ modulename ... ]" 291 exit 1 292 fi 293 294 for m in $* 295 do 296 echo inserting $m... 297 faulty_system modprobe $m 298 299 echo removing $m... 300 faulty_system modprobe -r $m 301 done 302 303------------------------------------------------------------------------------ 304 305- Inject page allocation failures only for a specific module:: 306 307 #!/bin/bash 308 309 FAILTYPE=fail_page_alloc 310 module=$1 311 312 if [ -z $module ] 313 then 314 echo "Usage: $0 <modulename>" 315 exit 1 316 fi 317 318 modprobe $module 319 320 if [ ! -d /sys/module/$module/sections ] 321 then 322 echo Module $module is not loaded 323 exit 1 324 fi 325 326 cat /sys/module/$module/sections/.text > /sys/kernel/debug/$FAILTYPE/require-start 327 cat /sys/module/$module/sections/.data > /sys/kernel/debug/$FAILTYPE/require-end 328 329 echo N > /sys/kernel/debug/$FAILTYPE/task-filter 330 echo 10 > /sys/kernel/debug/$FAILTYPE/probability 331 echo 100 > /sys/kernel/debug/$FAILTYPE/interval 332 echo -1 > /sys/kernel/debug/$FAILTYPE/times 333 echo 0 > /sys/kernel/debug/$FAILTYPE/space 334 echo 2 > /sys/kernel/debug/$FAILTYPE/verbose 335 echo 1 > /sys/kernel/debug/$FAILTYPE/ignore-gfp-wait 336 echo 1 > /sys/kernel/debug/$FAILTYPE/ignore-gfp-highmem 337 echo 10 > /sys/kernel/debug/$FAILTYPE/stacktrace-depth 338 339 trap "echo 0 > /sys/kernel/debug/$FAILTYPE/probability" SIGINT SIGTERM EXIT 340 341 echo "Injecting errors into the module $module... (interrupt to stop)" 342 sleep 1000000 343 344------------------------------------------------------------------------------ 345 346- Inject open_ctree error while btrfs mount:: 347 348 #!/bin/bash 349 350 rm -f testfile.img 351 dd if=/dev/zero of=testfile.img bs=1M seek=1000 count=1 352 DEVICE=$(losetup --show -f testfile.img) 353 mkfs.btrfs -f $DEVICE 354 mkdir -p tmpmnt 355 356 FAILTYPE=fail_function 357 FAILFUNC=open_ctree 358 echo $FAILFUNC > /sys/kernel/debug/$FAILTYPE/inject 359 printf %#x -12 > /sys/kernel/debug/$FAILTYPE/$FAILFUNC/retval 360 echo N > /sys/kernel/debug/$FAILTYPE/task-filter 361 echo 100 > /sys/kernel/debug/$FAILTYPE/probability 362 echo 0 > /sys/kernel/debug/$FAILTYPE/interval 363 echo -1 > /sys/kernel/debug/$FAILTYPE/times 364 echo 0 > /sys/kernel/debug/$FAILTYPE/space 365 echo 1 > /sys/kernel/debug/$FAILTYPE/verbose 366 367 mount -t btrfs $DEVICE tmpmnt 368 if [ $? -ne 0 ] 369 then 370 echo "SUCCESS!" 371 else 372 echo "FAILED!" 373 umount tmpmnt 374 fi 375 376 echo > /sys/kernel/debug/$FAILTYPE/inject 377 378 rmdir tmpmnt 379 losetup -d $DEVICE 380 rm testfile.img 381 382 383Tool to run command with failslab or fail_page_alloc 384---------------------------------------------------- 385In order to make it easier to accomplish the tasks mentioned above, we can use 386tools/testing/fault-injection/failcmd.sh. Please run a command 387"./tools/testing/fault-injection/failcmd.sh --help" for more information and 388see the following examples. 389 390Examples: 391 392Run a command "make -C tools/testing/selftests/ run_tests" with injecting slab 393allocation failure:: 394 395 # ./tools/testing/fault-injection/failcmd.sh \ 396 -- make -C tools/testing/selftests/ run_tests 397 398Same as above except to specify 100 times failures at most instead of one time 399at most by default:: 400 401 # ./tools/testing/fault-injection/failcmd.sh --times=100 \ 402 -- make -C tools/testing/selftests/ run_tests 403 404Same as above except to inject page allocation failure instead of slab 405allocation failure:: 406 407 # env FAILCMD_TYPE=fail_page_alloc \ 408 ./tools/testing/fault-injection/failcmd.sh --times=100 \ 409 -- make -C tools/testing/selftests/ run_tests 410 411Systematic faults using fail-nth 412--------------------------------- 413 414The following code systematically faults 0-th, 1-st, 2-nd and so on 415capabilities in the socketpair() system call:: 416 417 #include <sys/types.h> 418 #include <sys/stat.h> 419 #include <sys/socket.h> 420 #include <sys/syscall.h> 421 #include <fcntl.h> 422 #include <unistd.h> 423 #include <string.h> 424 #include <stdlib.h> 425 #include <stdio.h> 426 #include <errno.h> 427 428 int main() 429 { 430 int i, err, res, fail_nth, fds[2]; 431 char buf[128]; 432 433 system("echo N > /sys/kernel/debug/failslab/ignore-gfp-wait"); 434 sprintf(buf, "/proc/self/task/%ld/fail-nth", syscall(SYS_gettid)); 435 fail_nth = open(buf, O_RDWR); 436 for (i = 1;; i++) { 437 sprintf(buf, "%d", i); 438 write(fail_nth, buf, strlen(buf)); 439 res = socketpair(AF_LOCAL, SOCK_STREAM, 0, fds); 440 err = errno; 441 pread(fail_nth, buf, sizeof(buf), 0); 442 if (res == 0) { 443 close(fds[0]); 444 close(fds[1]); 445 } 446 printf("%d-th fault %c: res=%d/%d\n", i, atoi(buf) ? 'N' : 'Y', 447 res, err); 448 if (atoi(buf)) 449 break; 450 } 451 return 0; 452 } 453 454An example output:: 455 456 1-th fault Y: res=-1/23 457 2-th fault Y: res=-1/23 458 3-th fault Y: res=-1/12 459 4-th fault Y: res=-1/12 460 5-th fault Y: res=-1/23 461 6-th fault Y: res=-1/23 462 7-th fault Y: res=-1/23 463 8-th fault Y: res=-1/12 464 9-th fault Y: res=-1/12 465 10-th fault Y: res=-1/12 466 11-th fault Y: res=-1/12 467 12-th fault Y: res=-1/12 468 13-th fault Y: res=-1/12 469 14-th fault Y: res=-1/12 470 15-th fault Y: res=-1/12 471 16-th fault N: res=0/12 472