README.md
1JFuzz
2=====
3
4JFuzz is a tool for generating random programs with the objective
5of fuzz testing the ART infrastructure. Each randomly generated program
6can be run under various modes of execution, such as using the interpreter,
7using the optimizing compiler, using an external reference implementation,
8or using various target architectures. Any difference between the outputs
9(**divergence**) may indicate a bug in one of the execution modes.
10
11JFuzz can be combined with DexFuzz to get multi-layered fuzz testing.
12
13How to run JFuzz
14================
15
16 jfuzz [-s seed] [-d expr-depth] [-l stmt-length]
17 [-i if-nest] [-n loop-nest] [-v] [-h]
18
19where
20
21 -s : defines a deterministic random seed
22 (randomized using time by default)
23 -d : defines a fuzzing depth for expressions
24 (higher values yield deeper expressions)
25 -l : defines a fuzzing length for statement lists
26 (higher values yield longer statement sequences)
27 -i : defines a fuzzing nest for if/switch statements
28 (higher values yield deeper nested conditionals)
29 -n : defines a fuzzing nest for for/while/do-while loops
30 (higher values yield deeper nested loops)
31 -t : defines a fuzzing nest for try-catch-finally blocks
32 (higher values yield deeper nested try-catch-finally blocks)
33 -v : prints version number and exits
34 -h : prints help and exits
35
36The current version of JFuzz sends all output to stdout, and uses
37a fixed testing class named Test. So a typical test run looks as follows.
38
39 jfuzz > Test.java
40 mkdir classes
41 javac -d classes Test.java
42 dx --dex --output=classes.dex classes
43 art -cp classes.dex Test
44
45How to start JFuzz testing
46==========================
47
48 run_jfuzz_test.py
49 [--num_tests=NUM_TESTS]
50 [--device=DEVICE]
51 [--mode1=MODE] [--mode2=MODE]
52 [--report_script=SCRIPT]
53 [--jfuzz_arg=ARG]
54 [--true_divergence]
55 [--dexer=DEXER]
56 [--debug_info]
57
58where
59
60 --num_tests : number of tests to run (10000 by default)
61 --device : target device serial number (passed to adb -s)
62 --mode1 : m1
63 --mode2 : m2, with m1 != m2, and values one of
64 ri = reference implementation on host (default for m1)
65 hint = Art interpreter on host
66 hopt = Art optimizing on host (default for m2)
67 tint = Art interpreter on target
68 topt = Art optimizing on target
69 --report_script : path to script called for each divergence
70 --jfuzz_arg : argument for jfuzz
71 --true_divergence : don't bisect timeout divergences
72 --dexer=DEXER : use either dx or d8 to obtain dex files
73 --debug_info : include debugging info
74
75How to start JFuzz nightly testing
76==================================
77
78 run_jfuzz_test_nightly.py
79 [--num_proc NUM_PROC]
80
81where
82
83 --num_proc : number of run_jfuzz_test.py instances to run (8 by default)
84
85Remaining arguments are passed to run\_jfuzz_test.py.
86
87How to start J/DexFuzz testing (multi-layered)
88==============================================
89
90 run_dex_fuzz_test.py
91 [--num_tests=NUM_TESTS]
92 [--num_inputs=NUM_INPUTS]
93 [--device=DEVICE]
94 [--dexer=DEXER]
95 [--debug_info]
96
97where
98
99 --num_tests : number of tests to run (10000 by default)
100 --num_inputs : number of JFuzz programs to generate
101 --device : target device serial number (passed to adb -s)
102 --dexer=DEXER : use either dx or d8 to obtain dex files
103 --debug_info : include debugging info
104
105Background
106==========
107
108Although test suites are extremely useful to validate the correctness of a
109system and to ensure that no regressions occur, any test suite is necessarily
110finite in size and scope. Tests typically focus on validating particular
111features by means of code sequences most programmers would expect. Regression
112tests often use slightly less idiomatic code sequences, since they reflect
113problems that were not anticipated originally, but occurred “in the field”.
114Still, any test suite leaves the developer wondering whether undetected bugs
115and flaws still linger in the system.
116
117Over the years, fuzz testing has gained popularity as a testing technique for
118discovering such lingering bugs, including bugs that can bring down a system
119in an unexpected way. Fuzzing refers to feeding a large amount of random data
120as input to a system in an attempt to find bugs or make it crash. Generation-
121based fuzz testing constructs random, but properly formatted input data.
122Mutation-based fuzz testing applies small random changes to existing inputs
123in order to detect shortcomings in a system. Profile-guided or coverage-guided
124fuzzing adds a direction to the way these random changes are applied. Multi-
125layered approaches generate random inputs that are subsequently mutated at
126various stages of execution.
127
128The randomness of fuzz testing implies that the size and scope of testing is no
129longer bounded. Every new run can potentially discover bugs and crashes that were
130hereto undetected.
131