1Virtio-GPU Venus 2================ 3 4Venus is a Virtio-GPU protocol for Vulkan command serialization. The protocol 5definition and codegen are hosted at `venus-protocol 6<https://gitlab.freedesktop.org/olv/venus-protocol>`__. The renderer is 7hosted at `virglrenderer 8<https://gitlab.freedesktop.org/virgl/virglrenderer>`__. 9 10The protocol is still under development. This driver and the renderer are 11both considered experimental. 12 13Requirements 14------------ 15 16The Venus renderer requires 17 18- Vulkan 1.1 19- ``VK_EXT_external_memory_dma_buf`` 20- ``VK_EXT_image_drm_format_modifier`` 21- ``VK_EXT_queue_family_foreign`` 22 23from the host driver. However, it violates the spec in some places currently 24and also relies on implementation-defined behaviors in others. It is not 25expected to work on all drivers meeting the requirements. It has only been 26tested with 27 28- ANV 21.1 or later 29- RADV 21.1 or later (the host kernel must have 30 ``CONFIG_TRANSPARENT_HUGEPAGE`` disabled because of this `KVM issue 31 <https://github.com/google/security-research/security/advisories/GHSA-7wq5-phmq-m584>`__) 32- TURNIP 22.0 or later 33- Mali r32p0 or later 34 35The Venus driver requires supports for 36 37- ``VIRTGPU_PARAM_RESOURCE_BLOB`` 38- ``VIRTGPU_PARAM_HOST_VISIBLE`` 39- ``VIRTGPU_PARAM_CROSS_DEVICE`` 40- ``VIRTGPU_PARAM_CONTEXT_INIT`` 41 42from the virtio-gpu kernel driver, unless vtest is used. That usually means 43the guest kernel should be at least 5.16 or have the parameters back ported, 44paired with hypervisors such as `crosvm 45<https://chromium.googlesource.com/chromiumos/platform/crosvm>`__, or `patched 46qemu 47<https://www.collabora.com/news-and-blog/blog/2021/11/26/venus-on-qemu-enabling-new-virtual-vulkan-driver/>`__. 48 49vtest 50----- 51 52The simplest way to test Venus is to use virglrenderer's vtest server. To 53build virglrenderer with Venus support and to start the vtest server, 54 55.. code-block:: console 56 57 $ git clone https://gitlab.freedesktop.org/virgl/virglrenderer.git 58 $ cd virglrenderer 59 $ meson out -Dvenus-experimental=true 60 $ ninja -C out 61 $ ./out/vtest/virgl_test_server --venus 62 63In another shell, 64 65.. code-block:: console 66 67 $ export VK_ICD_FILENAMES=<path-to-virtio_icd.x86_64.json> 68 $ export VN_DEBUG=vtest 69 $ vulkaninfo 70 $ vkcube 71 72If the host driver of the system is not new enough, it is a good idea to build 73the host driver as well when building the Venus driver. Just remember to set 74:envvar:`VK_ICD_FILENAMES` when starting the vtest server so that the vtest 75server finds the locally built host driver. 76 77Virtio-GPU 78---------- 79 80The driver requires ``VIRTGPU_PARAM_CONTEXT_INIT`` from the virtio-gpu kernel 81driver, which was upstreamed in kernel 5.16. 82 83crosvm is written in Rust. To build crosvm, make sure Rust has been installed 84and 85 86.. code-block:: console 87 88 $ git clone --recurse-submodules \ 89 https://chromium.googlesource.com/chromiumos/platform/crosvm 90 $ cd crosvm 91 $ RUSTFLAGS="-L<path-to-virglrenderer>/out/src" cargo build \ 92 --features "x wl-dmabuf virgl_renderer virgl_renderer_next default-no-sandbox" 93 94Note that crosvm must be built with ``default-no-sandbox`` or started with 95``--disable-sandbox`` in this setup. 96 97This is how one might want to start crosvm 98 99.. code-block:: console 100 101 $ sudo LD_LIBRARY_PATH=<...> VK_ICD_FILENAMES=<...> ./target/debug/crosvm run \ 102 --gpu vulkan=true \ 103 --display-window-keyboard \ 104 --display-window-mouse \ 105 --host_ip 192.168.0.1 \ 106 --netmask 255.255.255.0 \ 107 --mac 12:34:56:78:9a:bc \ 108 --rwdisk disk.img \ 109 -p root=/dev/vda1 \ 110 <path-to-bzImage> 111 112assuming a working system is installed to partition 1 of ``disk.img``. 113``sudo`` or ``CAP_NET_ADMIN`` is needed to set up the TAP network device. 114 115Virtio-GPU and Virtio-WL 116------------------------ 117 118In this setup, the guest userspace uses Xwayland and a special Wayland 119compositor to connect guest X11/Wayland clients to the host Wayland 120compositor, using Virtio-WL as the transport. This setup is more tedious, but 121that should hopefully change over time. 122 123For now, the guest kernel must be built from the ``chromeos-5.10`` branch of 124the `Chrome OS kernel 125<https://chromium.googlesource.com/chromiumos/third_party/kernel>`__. 126 127To build minigbm and to enable minigbm support in virglrenderer, 128 129.. code-block:: console 130 131 $ git clone https://chromium.googlesource.com/chromiumos/platform/minigbm 132 $ cd minigbm 133 $ CFLAGS=-DDRV_<I915-or-your-driver> OUT=out DESTDIR=out/install make install 134 $ cd ../virglrenderer 135 $ meson configure out -Dminigbm_allocation=true 136 $ ninja -C out 137 138Make sure a host Wayland compositor is running. Replace 139``--display-window-keyboard --display-window-mouse`` by 140``--wayland-sock=<path-to-wayland-socket>`` when starting crosvm. 141 142In the guest, build and start sommelier, the special Wayland compositor, 143 144.. code-block:: console 145 146 $ git clone https://chromium.googlesource.com/chromiumos/platform2 147 $ cd platform2/vm_tools/sommelier 148 $ meson out -Dxwayland_path=/usr/bin/Xwayland -Dxwayland_gl_driver_path=/usr/lib/dri 149 $ ninja -C out 150 $ sudo chmod 777 /dev/wl0 151 $ ./out/sommelier -X --glamor 152 --xwayland-gl-driver-path=<path-to-locally-built-gl-driver> \ 153 sleep infinity 154 155Optional Requirements 156--------------------- 157 158When virglrenderer is built with ``-Dminigbm_allocation=true``, the Venus 159renderer might need to import GBM BOs. The imports will fail unless the host 160driver supports the formats, especially multi-planar ones, and the DRM format 161modifiers of the GBM BOs. 162 163In the future, if virglrenderer's ``virgl_renderer_export_fence`` is 164supported, the Venus renderer will require ``VK_KHR_external_fence_fd`` with 165``VK_EXTERNAL_FENCE_HANDLE_TYPE_SYNC_FD_BIT`` from the host driver. 166 167VK_MEMORY_PROPERTY_HOST_VISIBLE_BIT 168----------------------------------- 169 170The Venus renderer makes assumptions about ``VkDeviceMemory`` that has 171``VK_MEMORY_PROPERTY_HOST_VISIBLE_BIT``. The assumptions are illegal and rely 172on the current behaviors of the host drivers. It should be possible to remove 173some of the assumptions and incrementally improve compatibilities with more 174host drivers by imposing platform-specific requirements. But the long-term 175plan is to create a new Vulkan extension for the host drivers to address this 176specific use case. 177 178The Venus renderer assumes a device memory that has 179``VK_MEMORY_PROPERTY_HOST_VISIBLE_BIT`` can be exported as a mmapable dma-buf 180(in the future, the plan is to export the device memory as an opaque fd). It 181chains ``VkExportMemoryAllocateInfo`` to ``VkMemoryAllocateInfo`` without 182checking if the host driver can export the device memory. 183 184The dma-buf is mapped (in the future, the plan is to import the opaque fd and 185call ``vkMapMemory``) but the mapping is not accessed. Instead, the mapping 186is passed to ``KVM_SET_USER_MEMORY_REGION``. The hypervisor, host KVM, and 187the guest kernel work together to set up a write-back or write-combined guest 188mapping (see ``virtio_gpu_vram_mmap`` of the virtio-gpu kernel driver). CPU 189accesses to the device memory are via the guest mapping, and are assumed to be 190coherent when the device memory also has 191``VK_MEMORY_PROPERTY_HOST_COHERENT_BIT``. 192 193While the Venus renderer can force a ``VkDeviceMemory`` external, it does not 194force a ``VkImage`` or a ``VkBuffer`` external. As a result, it can bind an 195external device memory to a non-external resource. 196