1# Setup: Ubuntu host, Odroid C2 board, arm64 kernel 2 3These are the instructions on how to fuzz the kernel on an [Odroid C2](http://www.hardkernel.com/main/products/prdt_info.php) board using Ubuntu 14.04 on the host machine and Ubuntu on the Odroid. 4 5## Hardware setup 6 7### Required hardware 8 9Your hardware setup must satisfy the following requirements: 10 111. Host machine should be able to read the Odroid kernel log. 122. Host machine should be able to ssh to the Odroid board. 133. Host machine should be able to forcefully reboot the Odroid. 14 15The particular setup described below requires the following hardware: 16 171. [Odroid C2 board](http://www.hardkernel.com/main/products/prdt_info.php) 182. SD card (8 GB should be enough) 193. SD card reader (like [this one](https://www.amazon.de/gp/product/B009D79VH4/ref=oh_aui_detailpage_o06_s00?ie=UTF8&psc=1)) 204. [USB-UART cable](http://www.hardkernel.com/main/products/prdt_info.php?g_code=G134111883934) 215. USB Ethernet adapter (like [this one](https://www.amazon.de/Apple-MC704LL-A-USB-Ethernet-Adapter/dp/B00W7W9FK0/ref=dp_ob_title_ce)) 226. Ethernet cable 237. USB hub with [Per Port Power Switching support](http://www.gniibe.org/development/ac-power-control-by-USB-hub/index.html) (like D-Link DUB H7, **silver** edition). 248. [USB-DC Plug Cable](http://www.hardkernel.com/main/products/prdt_info.php?g_code=G141637559827) 25 26If you decide to use a different setup, you will need to update [Odroid-related code](https://github.com/google/syzkaller/blob/master/vm/odroid/odroid.go) in syzkaller manager. 27 28### Setup Odroid 29 301. Download and flash [Ubuntu image](http://odroid.com/dokuwiki/doku.php?id=en:c2_release_linux_ubuntu) onto SD card as described [here](http://odroid.com/dokuwiki/doku.php?id=en:odroid_flashing_tools). 312. Connect USB-UART cable and install minicom as described [here](http://odroid.com/dokuwiki/doku.php?id=en:usb_uart_kit). 323. Connect power plug, Odroid will start booting, make sure you see bootloader and kernel logs in minicom. 334. Make sure you can login through minicom as user `odroid` with password `odroid`. This user is a sudoer. 34 35When `systemd` starts Odroid stops sending kernel logs to UART. 36To fix this login to the Odroid board and add `kernel.printk = 7 4 1 3` line to `/etc/sysctl.conf` and then do `sysctl -p`: 37``` bash 38$ cat /etc/sysctl.conf | tail -n 1 39kernel.printk = 7 4 1 3 40$ sudo sysctl -p 41kernel.printk = 7 4 1 3 42``` 43 44Now make sure you can see kernel messages in minicom: 45``` 46$ echo "Some message" | sudo tee /dev/kmsg 47Some message 48[ 233.128597] Some message 49``` 50 51### Setup network 52 531. Connect USB Ethernet adapter to the host machine. 542. Use Ethernet cable to connect Odroid and the host adapter. 553. Use minicom to modify `/etc/network/interfaces` on Odroid: 56 57 ``` 58 auto eth0 59 iface eth0 inet static 60 address 172.16.0.31 61 gateway 172.16.0.1 62 netmask 255.255.255.0 63 ``` 64 654. Reboot Odroid. 66 675. Setup the interface on the host machine (though Network Manager or via `/etc/network/interfaces`): 68 69 ``` 70 auto eth1 71 iface eth1 inet static 72 address 172.16.0.30 73 gateway 172.16.0.1 74 netmask 255.255.255.0 75 ``` 76 776. You should now be able to ssh to Odroid (user `root`, password `odroid`): 78 79 ``` bash 80 $ ssh root@172.16.0.31 81 root@172.16.0.31's password: 82 ... 83 Last login: Thu Feb 11 11:30:51 2016 84 root@odroid64:~# 85 ``` 86 87### Setup USB hub 88 89To perform a hard reset of the Odroid board (by turning off power) I used a D-Link DUB H7 USB hub (**silver** edition, not the black one). 90This hub has support for a feature called [Per Port Power Switching](http://www.gniibe.org/development/ac-power-control-by-USB-hub/index.html), which allows to turn off power on a selected port on the hub remotely (via USB connection to the host machine) . 91 92[To be able to open the hub device entry](http://www.janosgyerik.com/adding-udev-rules-for-usb-debugging-android-devices/) under `/dev/` without being root, add the following file to `/etc/udev/rules.d/` on the host machine: 93``` bash 94$ cat /etc/udev/rules.d/10-local.rules 95SUBSYSTEM=="usb", ATTR{idVendor}=="2001", ATTR{idProduct}=="f103", MODE="0664", GROUP="plugdev" 96``` 97 98`idVendor` and `idProduct` should correspond to the hub vendor and product id (can be seen via `lsusb`). 99Don't forget to replug the hub after you add this file. 100 101``` bash 102$ lsusb 103... 104Bus 003 Device 026: ID 2001:f103 D-Link Corp. DUB-H7 7-port USB 2.0 hub 105... 106``` 107 108Communication with the hub is done by sending USB control messages, which requires `libusb`: 109``` bash 110sudo apt-get install libusb-dev libusb-1.0-0-dev 111``` 112 113Now plug in the hub and try to switch power on some of it's ports. 114For that you can use the [hub-ctrl.c](https://github.com/codazoda/hub-ctrl.c) tool by Niibe Yutaka or it's [ simplified Go analog](https://gist.github.com/xairy/37264952ff35da6e7dcf51ef486368e5): 115``` bash 116$ go run hub.go -bus=3 -device=26 -port=6 -power=0 117Power turned off on port 6 118$ go run hub.go -bus=3 -device=26 -port=6 -power=1 119Power turned on on port 6 120``` 121 122Note, that the DUB-H7 hub has a weird port numbering: `5, 6, 1, 2, 7, 3, 4` from left to right. 123 124Connect the Odroid board with a power plug to one of the USB hub ports and make sure you can forcefully reboot the Odroid by turning the power off and back on on this port. 125 126## Cross-compiler 127 128You need to compile full GCC cross-compiler tool-chain for aarch64 as described [here](http://preshing.com/20141119/how-to-build-a-gcc-cross-compiler/) (including the standard libraries). 129Use GCC revision 242378 (newer revisions should work as well, but weren't tested). 130The result should be a `$PREFIX` directory with cross-compiler, standard library headers, etc. 131``` 132$ ls $PREFIX 133aarch64-linux bin include lib libexec share 134``` 135 136## Kernel 137 138Set environment variables, they will be detected and used during kernel compilation: 139``` bash 140export PATH="$PREFIX/bin:$PATH" 141export ARCH=arm64 142export CROSS_COMPILE=aarch64-linux- 143``` 144 145Clone the linux-next kernel into `$KERNEL`: 146``` bash 147git clone https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git $KERNEL 148cd $KERNEL 149``` 150 151Apply the following patch, otherwise building the kernel with newer GCC fails (the patch is taken from [here](https://patchwork.kernel.org/patch/9380181/)): 152``` makefile 153diff --git a/Makefile b/Makefile 154index 165cf9783a5d..ff8b40dca9e2 100644 155--- a/Makefile 156+++ b/Makefile 157@@ -653,6 +653,11 @@ KBUILD_CFLAGS += $(call cc-ifversion, -lt, 0409, \ 158 # Tell gcc to never replace conditional load with a non-conditional one 159 KBUILD_CFLAGS += $(call cc-option,--param=allow-store-data-races=0) 160 161+# Stop gcc from converting switches into a form that defeats dead code 162+# elimination and can subsequently lead to calls to intentionally 163+# undefined functions appearing in the final link. 164+KBUILD_CFLAGS += $(call cc-option,--param=max-fsm-thread-path-insns=1) 165+ 166 include scripts/Makefile.gcc-plugins 167 168 ifdef CONFIG_READABLE_ASM 169``` 170 171Apply the following patch to disable KASAN bug detection on stack and globals (kernel doesn't boot, KASAN needs to be fixed): 172``` makefile 173diff --git a/scripts/Makefile.kasan b/scripts/Makefile.kasan 174index 9576775a86f6..8bc4eb36fc1b 100644 175--- a/scripts/Makefile.kasan 176+++ b/scripts/Makefile.kasan 177@@ -11,7 +11,6 @@ CFLAGS_KASAN_MINIMAL := -fsanitize=kernel-address 178 179 CFLAGS_KASAN := $(call cc-option, -fsanitize=kernel-address \ 180 -fasan-shadow-offset=$(KASAN_SHADOW_OFFSET) \ 181- --param asan-stack=1 --param asan-globals=1 \ 182 --param asan-instrumentation-with-call-threshold=$(call_threshold)) 183 184 ifeq ($(call cc-option, $(CFLAGS_KASAN_MINIMAL) -Werror),) 185``` 186 187Configure the kernel (you might wan't to enable more configs as listed [here](kernel_configs.md)): 188``` bash 189make defconfig 190# Edit .config to enable the following configs: 191# CONFIG_KCOV=y 192# CONFIG_KASAN=y 193# CONFIG_KASAN_INLINE=y 194# CONFIG_TEST_KASAN=m 195# CONFIG_PANIC_ON_OOPS=y 196make oldconfig 197``` 198 199Build the kernel: 200``` bash 201make -j48 dtbs Image modules LOCALVERSION=-xc2 202``` 203 204## Installation 205 206Install the `mkimage` util with arm64 support (part of the `u-boot-tools` package). 207You might have it by default, but it's not available on Ubuntu 14.04 in the default package repos. 208In this case download the package from [here](https://launchpad.net/ubuntu/xenial/amd64/u-boot-tools/2016.01+dfsg1-2ubuntu1) and use `sudo dpkg -i` to install. 209 210Insert the SD card reader with the SD card inside into the host machine. 211You should see two partitions automounted (or mount them manually), for example `sdb1` mounted at `$MOUNT_PATH/boot` and `sdb2` mounted at `$MOUNT_PATH/rootfs`. 212 213Build the kernel image: 214``` bash 215mkimage -A arm64 -O linux -T kernel -C none -a 0x1080000 -e 0x1080000 -n linux-next -d arch/arm64/boot/Image ./uImage 216``` 217 218Copy the kernel image, modules and device tree: 219``` bash 220KERNEL_VERSION=`cat ./include/config/kernel.release` 221cp ./uImage $MOUNT_PATH/boot/uImage-$KERNEL_VERSION 222make modules_install LOCALVERSION=-xc2 INSTALL_MOD_PATH=$MOUNT_PATH/rootfs/ 223cp ./arch/arm64/boot/dts/amlogic/meson-gxbb-odroidc2.dtb $MOUNT_PATH/boot/meson-gxbb-odroidc2-$KERNEL_VERSION.dtb 224cp .config $MOUNT_PATH/boot/config-$KERNEL_VERSION 225``` 226 227Backup the old bootloader config; if something doesn't work with the new kernel, you can always roll back to the old one by restoring `boot.ini`: 228``` bash 229cd $MOUNT_PATH/boot/ 230cp boot.ini boot.ini.orig 231``` 232 233Replace the bootloader config `boot.ini` (based on the one taken from [here](http://forum.odroid.com/viewtopic.php?p=162045#p162045)) with the following; don't forget to update `version`: 234``` 235ODROIDC2-UBOOT-CONFIG 236 237# Set version to $KERNEL_VERSION 238setenv version 4.11.0-rc1-next-20170308-xc2-dirty 239setenv uImage uImage-${version} 240setenv fdtbin meson-gxbb-odroidc2-${version}.dtb 241 242setenv initrd_high 0xffffffff 243setenv fdt_high 0xffffffff 244setenv uimage_addr_r 0x01080000 245setenv fdtbin_addr_r 0x01000000 246 247# You might need to use root=/dev/mmcblk0p2 below, try booting and see if the current one works. 248setenv bootargs "console=ttyAML0,115200 root=/dev/mmcblk1p2 rootwait ro fsck.mode=force fsck.repair=yes net.ifnames=0 oops=panic panic_on_warn=1 panic=86400 systemd.show_status=no" 249 250fatload mmc 0:1 ${fdtbin_addr_r} ${fdtbin} 251fatload mmc 0:1 ${uimage_addr_r} ${uImage} 252bootm ${uimage_addr_r} - ${fdtbin_addr_r} 253``` 254 255Sync and unmount: 256``` bash 257sync 258umount $MOUNT_PATH/boot 259umount $MOUNT_PATH/rootfs 260``` 261 262Now plug the SD card into the Odroid board and boot. 263The new kernel should now be used. 264It makes sense to ensure that you still can ssh to Odroid. 265 266## Syzkaller 267 268Generate ssh key and copy it to Odroid: 269``` bash 270mkdir ssh 271ssh-keygen -f ssh/id_rsa -t rsa -N '' 272ssh root@172.16.0.31 "mkdir /root/.ssh/" 273scp ./ssh/id_rsa.pub root@172.16.0.31:/root/.ssh/authorized_keys 274``` 275 276Now make sure you can ssh with the key: 277``` bash 278ssh -i ./ssh/id_rsa root@172.16.0.31 279``` 280 281Build syzkaller with `odroid` build tag: 282``` bash 283make GOTAGS=odroid TARGETARCH=arm64 284``` 285 286Use the following config: 287``` 288{ 289 "target": "linux/arm64", 290 "http": "127.0.0.1:56741", 291 "workdir": "/syzkaller/workdir", 292 "kernel_obj": "/linux-next", 293 "syzkaller": "/go/src/github.com/google/syzkaller", 294 "sshkey": "/odroid/ssh/id_rsa", 295 "rpc": "172.16.0.30:0", 296 "sandbox": "namespace", 297 "reproduce": false, 298 "procs": 8, 299 "type": "odroid", 300 "vm": { 301 "host_addr": "172.16.0.30", 302 "slave_addr": "172.16.0.31", 303 "console": "/dev/ttyUSB0", 304 "hub_bus": 3, 305 "hub_device": 26, 306 "hub_port": 5 307 } 308} 309``` 310 311Don't forget to update: 312 - `workdir` (path to the workdir) 313 - `kernel_obj` (path to kernel build directory) 314 - `sshkey` (path to the generated ssh private key) 315 - `vm.console` (serial device you used in `minicom`) 316 - `vm.hub_bus` (number of the bus to which USB hub is connected, view with `lsusb`) 317 - `vm.hub_device` (device number for the USB hub, view with `lsusb`) 318 - `vm.hub_port` (number of the USB hub port to which Odroid power plug is connected) 319 320Now start syzkaller: 321``` bash 322./bin/syz-manager -config=odroid.cfg 323``` 324 325If you get issues after `syz-manager` starts, consider running it with the `-debug` flag. 326Also see [this page](/docs/troubleshooting.md) for troubleshooting tips. 327