1Virtio-GPU Venus 2================ 3 4Venus is a Virtio-GPU protocol for Vulkan command serialization. The protocol 5definition and codegen are hosted at `venus-protocol 6<https://gitlab.freedesktop.org/olv/venus-protocol>`__. The renderer is 7hosted at `virglrenderer 8<https://gitlab.freedesktop.org/virgl/virglrenderer>`__. 9 10The protocol is still under development. This driver and the renderer are 11both considered experimental. 12 13Requirements 14------------ 15 16The Venus renderer requires 17 18- Vulkan 1.1 19- ``VK_EXT_external_memory_dma_buf`` 20- ``VK_EXT_image_drm_format_modifier`` 21- ``VK_EXT_queue_family_foreign`` 22 23from the host driver. However, it violates the spec in some places currently 24and also relies on implementation-defined behaviors in others. It is not 25expected to work on all drivers meeting the requirements. It has only been 26tested with 27 28- ANV 21.1 or later 29- RADV 21.1 or later (the host kernel must have 30 ``CONFIG_TRANSPARENT_HUGEPAGE`` disabled because of this `KVM issue 31 <https://github.com/google/security-research/security/advisories/GHSA-7wq5-phmq-m584>`__) 32 33The Venus driver requires supports for 34 35- ``VIRTGPU_PARAM_RESOURCE_BLOB`` 36- ``VIRTGPU_PARAM_HOST_VISIBLE`` 37- ``VIRTGPU_PARAM_CROSS_DEVICE`` 38- ``VIRTGPU_PARAM_CONTEXT_INIT`` 39 40from the virtio-gpu kernel driver, unless vtest is used. Currently, this 41means the `context-init 42<https://gitlab.freedesktop.org/virgl/drm-misc-next/-/tree/context-init>`__ 43kernel branch paired with `crosvm 44<https://chromium.googlesource.com/chromiumos/platform/crosvm>`__. 45 46vtest 47----- 48 49The simplest way to test Venus is to use virglrenderer's vtest server. To 50build virglrenderer with Venus support and to start the vtest server, 51 52.. code-block:: console 53 54 $ git clone https://gitlab.freedesktop.org/virgl/virglrenderer.git 55 $ cd virglrenderer 56 $ meson out -Dvenus-experimental=true 57 $ ninja -C out 58 $ ./out/vtest/virgl_test_server --venus 59 60In another shell, 61 62.. code-block:: console 63 64 $ export VK_ICD_FILENAMES=<path-to-virtio_icd.x86_64.json> 65 $ export VN_DEBUG=vtest 66 $ vulkaninfo 67 $ vkcube 68 69If the host driver of the system is not new enough, it is a good idea to build 70the host driver as well when building the Venus driver. Just remember to set 71:envvar:`VK_ICD_FILENAMES` when starting the vtest server so that the vtest 72server finds the locally built host driver. 73 74Virtio-GPU 75---------- 76 77Because the driver requires ``VIRTGPU_PARAM_CONTEXT_INIT`` from the virtio-gpu 78kernel driver, one must make sure the guest kernel includes the changes from 79the `context-init 80<https://gitlab.freedesktop.org/virgl/drm-misc-next/-/tree/context-init>`__ 81branch. 82 83To build crosvm, 84 85.. code-block:: console 86 87 $ mkdir crosvm 88 $ cd crosvm 89 $ wget https://storage.googleapis.com/git-repo-downloads/repo 90 $ chmod +x repo 91 $ ./repo init -g crosvm -u https://chromium.googlesource.com/chromiumos/manifest.git 92 $ ./repo sync 93 $ cd src/platform/crosvm 94 $ RUSTFLAGS="-L<path-to-virglrenderer>/out/src" cargo build \ 95 --features "x virgl_renderer virgl_renderer_next default-no-sandbox" 96 97Note that crosvm must be built with ``default-no-sandbox`` or started with 98``--disable-sandbox`` in this setup. 99 100This is how one might want to start crosvm 101 102.. code-block:: console 103 104 $ sudo LD_LIBRARY_PATH=<...> VK_ICD_FILENAMES=<...> ./target/debug/crosvm run \ 105 --gpu vulkan=true \ 106 --display-window-keyboard \ 107 --display-window-mouse \ 108 --host_ip 192.168.0.1 \ 109 --netmask 255.255.255.0 \ 110 --mac 12:34:56:78:9a:bc \ 111 --rwdisk disk.qcow2 \ 112 -p root=/dev/vda1 \ 113 <path-to-bzImage> 114 115assuming a working system is installed to partition 1 of ``disk.qcow2``. 116``sudo`` or ``CAP_NET_ADMIN`` is needed to set up the TAP network device. 117 118Virtio-GPU and Virtio-WL 119------------------------ 120 121In this setup, the guest userspace uses Xwayland and a special Wayland 122compositor to connect guest X11/Wayland clients to the host Wayland 123compositor, using Virtio-WL as the transport. This setup is more tedious, but 124that should hopefully change over time. 125 126For now, the guest kernel must be built from the ``chromeos-5.10`` branch of 127the `Chrome OS kernel 128<https://chromium.googlesource.com/chromiumos/third_party/kernel>`__. crosvm 129should also be built with ``wl-dmabuf`` feature rather than ``x`` feature. 130 131To build minigbm and to enable minigbm support in virglrenderer, 132 133.. code-block:: console 134 135 $ git clone https://chromium.googlesource.com/chromiumos/platform/minigbm 136 $ cd minigbm 137 $ CFLAGS=-DDRV_<I915-or-your-driver> OUT=out DESTDIR=out/install make install 138 $ cd ../virglrenderer 139 $ meson configure out -Dminigbm_allocation=true 140 $ ninja -C out 141 142Make sure a host Wayland compositor is running. Replace 143``--display-window-keyboard --display-window-mouse`` by 144``--wayland-sock=<path-to-wayland-socket>`` when starting crosvm. 145 146In the guest, build and start sommelier, the special Wayland compositor, 147 148.. code-block:: console 149 150 $ git clone https://chromium.googlesource.com/chromiumos/platform2 151 $ cd platform2/vm_tools/sommelier 152 $ meson out -Dxwayland_path=/usr/bin/Xwayland -Dxwayland_gl_driver_path=/usr/lib/dri 153 $ ninja -C out 154 $ sudo chmod 777 /dev/wl0 155 $ ./out/sommelier -X --glamor 156 --xwayland-gl-driver-path=<path-to-locally-built-gl-driver> \ 157 sleep infinity 158 159sommelier requires ``xdg-shell-unstable-v6`` rather than the stable 160``xdg-shell`` from the host compositor. One must make sure the host 161compositor still supports the older extension. 162 163Optional Requirements 164--------------------- 165 166When virglrenderer is built with ``-Dminigbm_allocation=true``, the Venus 167renderer might need to import GBM BOs. The imports will fail unless the host 168driver supports the formats, especially multi-planar ones, and the DRM format 169modifiers of the GBM BOs. 170 171In the future, if virglrenderer's ``virgl_renderer_export_fence`` is 172supported, the Venus renderer will require ``VK_KHR_external_fence_fd`` with 173``VK_EXTERNAL_FENCE_HANDLE_TYPE_SYNC_FD_BIT`` from the host driver. 174 175A WSI image of the Venus driver is an external image to the host driver. When 176the WSI image is transitioned from ``VK_IMAGE_LAYOUT_UNDEFINED`` after image 177acquisition, the Venus driver does not request the Venus renderer to perform 178an ownership transfer on the external image. It is unclear if the ownership 179transfer is required or not. A specification issue has been filed for 180clarifications. See the comment before ``vn_cmd_fix_image_memory_barrier`` 181for more details. 182 183VK_MEMORY_PROPERTY_HOST_VISIBLE_BIT 184----------------------------------- 185 186The Venus renderer makes assumptions about ``VkDeviceMemory`` that has 187``VK_MEMORY_PROPERTY_HOST_VISIBLE_BIT``. The assumptions are illegal and rely 188on the current behaviors of the host drivers. It should be possible to remove 189some of the assumptions and incrementally improve compatibilities with more 190host drivers by imposing platform-specific requirements. But the long-term 191plan is to create a new Vulkan extension for the host drivers to address this 192specific use case. 193 194The Venus renderer assumes a device memory that has 195``VK_MEMORY_PROPERTY_HOST_VISIBLE_BIT`` can be exported as a mmapable dma-buf 196(in the future, the plan is to export the device memory as an opaque fd). It 197chains ``VkExportMemoryAllocateInfo`` to ``VkMemoryAllocateInfo`` without 198checking if the host driver can export the device memory. 199 200The dma-buf is mapped (in the future, the plan is to import the opaque fd and 201call ``vkMapMemory``) but the mapping is not accessed. Instead, the mapping 202is passed to ``KVM_SET_USER_MEMORY_REGION``. The hypervisor, host KVM, and 203the guest kernel work together to set up a write-back or write-combined guest 204mapping (see ``virtio_gpu_vram_mmap`` of the virtio-gpu kernel driver). CPU 205accesses to the device memory are via the guest mapping, and are assumed to be 206coherent when the device memory also has 207``VK_MEMORY_PROPERTY_HOST_COHERENT_BIT``. 208 209When a ``VkImage`` or a ``VkBuffer`` is created, the Venus renderer does not 210know if the image or the buffer will be bound to such a device memory or not. 211As a result, the Venus renderer unconditionally chains 212``VkExternalMemoryImageCreateInfo`` to ``VkImageCreateInfo`` and chains 213``VkExternalMemoryBufferCreateInfo`` to ``VkBufferCreateInfo`` without 214checking for the host driver support. 215