1# Advanced Usage 2 3To see the usage information for your version of crosvm, run `crosvm` or `crosvm run --help`. 4 5## Boot a Kernel 6 7To run a very basic VM with just a kernel and default devices: 8 9```sh 10crosvm run "${KERNEL_PATH}" 11``` 12 13The uncompressed kernel image, also known as vmlinux, can be found in your kernel build directory in 14the case of x86 at `arch/x86/boot/compressed/vmlinux`. 15 16## Rootfs 17 18### With a disk image 19 20In most cases, you will want to give the VM a virtual block device to use as a root file system: 21 22```sh 23crosvm run -r "${ROOT_IMAGE}" "${KERNEL_PATH}" 24``` 25 26The root image must be a path to a disk image formatted in a way that the kernel can read. Typically 27this is a squashfs image made with `mksquashfs` or an ext4 image made with `mkfs.ext4`. By using the 28`-r` argument, the kernel is automatically told to use that image as the root, and therefore can 29only be given once. More disks can be given with `-d` or `--rwdisk` if a writable disk is desired. 30 31To run crosvm with a writable rootfs: 32 33> **WARNING:** Writable disks are at risk of corruption by a malicious or malfunctioning guest OS. 34 35```sh 36crosvm run --rwdisk "${ROOT_IMAGE}" -p "root=/dev/vda" vmlinux 37``` 38 39> **NOTE:** If more disks arguments are added prior to the desired rootfs image, the `root=/dev/vda` 40> must be adjusted to the appropriate letter. 41 42### With virtiofs 43 44Linux kernel 5.4+ is required for using virtiofs. This is convenient for testing. The file system 45must be named "mtd\*" or "ubi\*". 46 47```sh 48crosvm run --shared-dir "/:mtdfake:type=fs:cache=always" \ 49 -p "rootfstype=virtiofs root=mtdfake" vmlinux 50``` 51 52## Network device 53 54The most convenient way to provide a network device to a guest is to setup a persistent TAP 55interface on the host. This section will explain how to do this for basic IPv4 connectivity. 56 57```sh 58sudo ip tuntap add mode tap user $USER vnet_hdr crosvm_tap 59sudo ip addr add 192.168.10.1/24 dev crosvm_tap 60sudo ip link set crosvm_tap up 61``` 62 63These commands create a TAP interface named `crosvm_tap` that is accessible to the current user, 64configure the host to use the IP address `192.168.10.1`, and bring the interface up. 65 66The next step is to make sure that traffic from/to this interface is properly routed: 67 68```sh 69sudo sysctl net.ipv4.ip_forward=1 70# Network interface used to connect to the internet. 71HOST_DEV=$(ip route get 8.8.8.8 | awk -- '{printf $5}') 72sudo iptables -t nat -A POSTROUTING -o "${HOST_DEV}" -j MASQUERADE 73sudo iptables -A FORWARD -i "${HOST_DEV}" -o crosvm_tap -m state --state RELATED,ESTABLISHED -j ACCEPT 74sudo iptables -A FORWARD -i crosvm_tap -o "${HOST_DEV}" -j ACCEPT 75``` 76 77The interface is now configured and can be used by crosvm: 78 79```sh 80crosvm run \ 81 ... 82 --tap-name crosvm_tap \ 83 ... 84``` 85 86Provided the guest kernel had support for `VIRTIO_NET`, the network device should be visible and 87configurable from the guest: 88 89```sh 90# Replace with the actual network interface name of the guest 91# (use "ip addr" to list the interfaces) 92GUEST_DEV=enp0s5 93sudo ip addr add 192.168.10.2/24 dev "${GUEST_DEV}" 94sudo ip link set "${GUEST_DEV}" up 95sudo ip route add default via 192.168.10.1 96# "8.8.8.8" is chosen arbitrarily as a default, please replace with your local (or preferred global) 97# DNS provider, which should be visible in `/etc/resolv.conf` on the host. 98echo "nameserver 8.8.8.8" | sudo tee /etc/resolv.conf 99``` 100 101These commands assign IP address `192.168.10.2` to the guest, activate the interface, and route all 102network traffic to the host. The last line also ensures DNS will work. 103 104Please refer to your distribution's documentation for instructions on how to make these settings 105persistent for the host and guest if desired. 106 107## Control Socket 108 109If the control socket was enabled with `-s`, the main process can be controlled while crosvm is 110running. To tell crosvm to stop and exit, for example: 111 112> **NOTE:** If the socket path given is for a directory, a socket name underneath that path will be 113> generated based on crosvm's PID. 114 115```sh 116crosvm run -s /run/crosvm.sock ${USUAL_CROSVM_ARGS} 117 <in another shell> 118crosvm stop /run/crosvm.sock 119``` 120 121> **WARNING:** The guest OS will not be notified or gracefully shutdown. 122 123This will cause the original crosvm process to exit in an orderly fashion, allowing it to clean up 124any OS resources that might have stuck around if crosvm were terminated early. 125 126## Multiprocess Mode 127 128By default crosvm runs in multiprocess mode. Each device that supports running inside of a sandbox 129will run in a jailed child process of crosvm. The appropriate minijail seccomp policy files must be 130present either in `/usr/share/policy/crosvm` or in the path specified by the `--seccomp-policy-dir` 131argument. The sandbox can be disabled for testing with the `--disable-sandbox` option. 132 133## Wayland forwarding to host 134 135If you have a Wayland compositor running on your host, it is possible to display and control guest 136applications from it. This requires: 137 138- A guest kernel version 5.16 or above with `CONFIG_DRM_VIRTIO_GPU` enabled, 139- The `sommelier` Wayland proxy in your guest image. 140 141This section will walk you through the steps needed to get this to work. 142 143### Guest kernel requirements 144 145Wayland support on crosvm relies on virtio-gpu contexts, which have been introduced in Linux 5.16. 146Make sure your guest kernel is either this version or a more recent one, and that 147`CONFIG_DRM_VIRTIO_GPU` is enabled in your kernel configuration. 148 149### Crosvm requirements 150 151Wayland forwarding requires the GPU feature and any non-2d virtio-gpu mode to be enabled, so pass 152them to your `cargo build` or `cargo run` command, e.g: 153 154```sh 155cargo build --features "gpu,virgl_renderer,virgl_renderer_next" 156``` 157 158### Building sommelier 159 160[Sommelier] is a proxy Wayland compositor that forwards the Wayland protocol from a guest to a 161compositor running on the host through the guest GPU device. As it is not a standard tool, we will 162have to build it by ourselves. It is recommended to do this from the guest 163[with networking enabled](./example_usage.md#add-networking-support). 164 165Clone Chrome OS' `platform2` repository, which contains the source for sommelier: 166 167```sh 168git clone https://chromium.googlesource.com/chromiumos/platform2 169``` 170 171Go into the sommelier directory and prepare for building: 172 173```sh 174cd platform2/vm_tools/sommelier/ 175meson setup build -Dwith_tests=false 176``` 177 178This setup step will check for all libraries required to build sommelier. If some are missing, 179install them using your guest's distro package manager and re-run `meson setup` until it passes. 180 181Finally, build sommelier and install it: 182 183```sh 184meson compile -C build 185sudo meson install -C build 186``` 187 188This last step will put the `sommelier` binary into `/usr/local/bin`. 189 190### Running guest Wayland apps 191 192Crosvm can connect to a running Wayland server (e.g. [weston]) on the host and forward the protocol 193from all Wayland guest applications to it. To enable this you need to know the socket of the Wayland 194server running on your host - typically it would be `$XDG_RUNTIME_DIR/wayland-0`. 195 196Once you have confirmed the socket, create a GPU device and enable forwarding by adding the 197`--gpu --wayland-sock $XDG_RUNTIME_DIR/wayland-0` arguments to your crosvm command-line. 198 199You can now run Wayland clients through sommelier, e.g: 200 201```sh 202sommelier --virtgpu-channel weston-terminal 203``` 204 205Or 206 207```sh 208sommelier --virtgpu-channel gedit 209``` 210 211Applications started that way should appear on and be controllable from the Wayland server running 212on your host. 213 214The `--virtgpu-channel` option is currently necessary for sommelier to work with the setup of this 215document, but will likely not be required in the future. 216 217If you have `Xwayland` installed in the guest you can also run X applications: 218 219```sh 220sommelier -X --xwayland-path=/usr/bin/Xwayland xeyes 221``` 222 223## GDB Support 224 225crosvm supports [GDB Remote Serial Protocol] to allow developers to debug guest kernel via GDB. 226 227You can enable the feature by `--gdb` flag: 228 229```sh 230# Use uncompressed vmlinux 231crosvm run --gdb <port> ${USUAL_CROSVM_ARGS} vmlinux 232``` 233 234Then, you can start GDB in another shell. 235 236```sh 237gdb vmlinux 238(gdb) target remote :<port> 239(gdb) hbreak start_kernel 240(gdb) c 241<start booting in the other shell> 242``` 243 244For general techniques for debugging the Linux kernel via GDB, see this [kernel documentation]. 245 246## Defaults 247 248The following are crosvm's default arguments and how to override them. 249 250- 256MB of memory (set with `-m`) 251- 1 virtual CPU (set with `-c`) 252- no block devices (set with `-r`, `-d`, or `--rwdisk`) 253- no network (set with `--host_ip`, `--netmask`, and `--mac`) 254- only the kernel arguments necessary to run with the supported devices (add more with `-p`) 255- run in multiprocess mode (run in single process mode with `--disable-sandbox`) 256- no control socket (set with `-s`) 257 258[gdb remote serial protocol]: https://sourceware.org/gdb/onlinedocs/gdb/Remote-Protocol.html 259[kernel documentation]: https://www.kernel.org/doc/html/latest/dev-tools/gdb-kernel-debugging.html 260[sommelier]: https://chromium.googlesource.com/chromiumos/platform2/+/master/vm_tools/sommelier 261[weston]: https://github.com/wayland-project/weston 262