README.md
1JFuzz
2=====
3
4JFuzz is a tool for generating random programs with the objective
5of fuzz testing the ART infrastructure. Each randomly generated program
6can be run under various modes of execution, such as using the interpreter,
7using the optimizing compiler, using an external reference implementation,
8or using various target architectures. Any difference between the outputs
9(**divergence**) may indicate a bug in one of the execution modes.
10
11JFuzz can be combined with DexFuzz to get multi-layered fuzz testing.
12
13How to run JFuzz
14================
15
16 jfuzz [-s seed] [-d expr-depth] [-l stmt-length]
17 [-i if-nest] [-n loop-nest] [-v] [-h]
18
19where
20
21 -s : defines a deterministic random seed
22 (randomized using time by default)
23 -d : defines a fuzzing depth for expressions
24 (higher values yield deeper expressions)
25 -l : defines a fuzzing length for statement lists
26 (higher values yield longer statement sequences)
27 -i : defines a fuzzing nest for if/switch statements
28 (higher values yield deeper nested conditionals)
29 -n : defines a fuzzing nest for for/while/do-while loops
30 (higher values yield deeper nested loops)
31 -t : defines a fuzzing nest for try-catch-finally blocks
32 (higher values yield deeper nested try-catch-finally blocks)
33 -v : prints version number and exits
34 -h : prints help and exits
35
36The current version of JFuzz sends all output to stdout, and uses
37a fixed testing class named Test. So a typical test run looks as follows.
38
39 jfuzz > Test.java
40 jack -cp ${JACK_CLASSPATH} --output-dex . Test.java
41 art -classpath classes.dex Test
42
43How to start JFuzz testing
44==========================
45
46 run_jfuzz_test.py
47 [--num_tests=NUM_TESTS]
48 [--device=DEVICE]
49 [--mode1=MODE] [--mode2=MODE]
50 [--report_script=SCRIPT]
51 [--jfuzz_arg=ARG]
52 [--true_divergence]
53 [--dexer=DEXER]
54 [--debug_info]
55
56where
57
58 --num_tests : number of tests to run (10000 by default)
59 --device : target device serial number (passed to adb -s)
60 --mode1 : m1
61 --mode2 : m2, with m1 != m2, and values one of
62 ri = reference implementation on host (default for m1)
63 hint = Art interpreter on host
64 hopt = Art optimizing on host (default for m2)
65 tint = Art interpreter on target
66 topt = Art optimizing on target
67 --report_script : path to script called for each divergence
68 --jfuzz_arg : argument for jfuzz
69 --true_divergence : don't bisect timeout divergences
70 --dexer=DEXER : use either dx, d8, or jack to obtain dex files
71 --debug_info : include debugging info
72
73How to start JFuzz nightly testing
74==================================
75
76 run_jfuzz_test_nightly.py
77 [--num_proc NUM_PROC]
78
79where
80
81 --num_proc : number of run_jfuzz_test.py instances to run (8 by default)
82
83Remaining arguments are passed to run\_jfuzz_test.py.
84
85How to start J/DexFuzz testing (multi-layered)
86==============================================
87
88 run_dex_fuzz_test.py
89 [--num_tests=NUM_TESTS]
90 [--num_inputs=NUM_INPUTS]
91 [--device=DEVICE]
92 [--dexer=DEXER]
93 [--debug_info]
94
95where
96
97 --num_tests : number of tests to run (10000 by default)
98 --num_inputs : number of JFuzz programs to generate
99 --device : target device serial number (passed to adb -s)
100 --dexer=DEXER : use either dx, d8, or jack to obtain dex files
101 --debug_info : include debugging info
102
103Background
104==========
105
106Although test suites are extremely useful to validate the correctness of a
107system and to ensure that no regressions occur, any test suite is necessarily
108finite in size and scope. Tests typically focus on validating particular
109features by means of code sequences most programmers would expect. Regression
110tests often use slightly less idiomatic code sequences, since they reflect
111problems that were not anticipated originally, but occurred “in the field”.
112Still, any test suite leaves the developer wondering whether undetected bugs
113and flaws still linger in the system.
114
115Over the years, fuzz testing has gained popularity as a testing technique for
116discovering such lingering bugs, including bugs that can bring down a system
117in an unexpected way. Fuzzing refers to feeding a large amount of random data
118as input to a system in an attempt to find bugs or make it crash. Generation-
119based fuzz testing constructs random, but properly formatted input data.
120Mutation-based fuzz testing applies small random changes to existing inputs
121in order to detect shortcomings in a system. Profile-guided or coverage-guided
122fuzzing adds a direction to the way these random changes are applied. Multi-
123layered approaches generate random inputs that are subsequently mutated at
124various stages of execution.
125
126The randomness of fuzz testing implies that the size and scope of testing is no
127longer bounded. Every new run can potentially discover bugs and crashes that were
128hereto undetected.
129