// Copyright 2021-2024 The Khronos Group Inc. // // SPDX-License-Identifier: CC-BY-4.0 = VK_EXT_metal_objects :toc: left :refpage: https://registry.khronos.org/vulkan/specs/1.3-extensions/man/html/ :sectnums: This document details API design ideas for the `VK_EXT_metal_objects` extension, which, in a Vulkan implementation that is layered on top of Metal on Apple device platforms, provides the ability to import and export the underlying Metal objects associated with specific Vulkan objects. == Problem Statement In a Vulkan implementation that is layered on top of Metal on Apple device platforms, many Vulkan objects have a corresponding Metal equivalent that provides the underlying functionality. When integrating a Vulkan app onto such a platform, it is often useful for the app to be able to interact directly with these Metal objects: . Pass Metal texture and buffer resources between Vulkan and platform functionality such as camera, video, AR, or inter-process frameworks. . Create hybrid apps that combine and mingle Vulkan and Metal operations, by sharing, in addition to Metal texture and buffer resources, also Metal device, command queues, and synchronization operations. == Solution Space While the first use case listed above might be addressed via extensions built by adding Metal platform variations to the `VK_KHR_external_memory` family of extensions, the second use case requires a different approach. Rather than create a fragmented approach to managing access to Metal objects, this extension proposes to provide a single unified interface to all Metal objects providing functionality to the layered Vulkan implementation. Furthermore, this extension has been designed to be easily extendable, to import and export new Metal objects that might be introduced in the future. The intent is that this extension will be advertised and supported only on implementations that are layered on top of Metal on Apple device platforms. == Proposal === Exporting Metal Objects To export underlying Metal objects from Vulkan objects, this extension adds one new Vulkan command: [source,c] ---- VKAPI_ATTR void VKAPI_CALL vkExportMetalObjectsEXT( VkDevice device, VkExportMetalObjectsInfoEXT* pMetalObjectsInfo); ---- The `pNext` chain of the `VkExportMetalObjectsInfoEXT` in the `pMetalObjectsInfo` parameter can contain a variety of specialized structures, each targeted to identify a particular Vulkan object and a pointer through which the underlying Metal object can be exported, as identified in the following list. The use of `pNext` structures allows this extension to be flexibly extended in the future to permit access to additional Metal objects, through the future addition of new export object structures. [source,c] ---- typedef struct VkExportMetalObjectsInfoEXT { VkStructureType sType; const void* pNext; } VkExportMetalObjectsInfoEXT; typedef struct VkExportMetalDeviceInfoEXT { VkStructureType sType; const void* pNext; MTLDevice_id mtlDevice; } VkExportMetalDeviceInfoEXT; typedef struct VkExportMetalCommandQueueInfoEXT { VkStructureType sType; const void* pNext; VkQueue queue; MTLCommandQueue_id mtlCommandQueue; } VkExportMetalCommandQueueInfoEXT; typedef struct VkExportMetalBufferInfoEXT { VkStructureType sType; const void* pNext; VkDeviceMemory memory; MTLBuffer_id mtlBuffer; } VkExportMetalBufferInfoEXT; typedef struct VkExportMetalTextureInfoEXT { VkStructureType sType; const void* pNext; VkImage image; VkImageView imageView; VkBufferView bufferView; VkImageAspectFlagBits plane; MTLTexture_id mtlTexture; } VkExportMetalTextureInfoEXT; typedef struct VkExportMetalIOSurfaceInfoEXT { VkStructureType sType; const void* pNext; VkImage image; IOSurfaceRef ioSurface; } VkExportMetalIOSurfaceInfoEXT; typedef struct VkExportMetalSharedEventInfoEXT { VkStructureType sType; const void* pNext; VkSemaphore semaphore; VkEvent event; MTLSharedEvent_id mtlSharedEvent; } VkExportMetalSharedEventInfoEXT; ---- To export Metal objects from Vulkan objects, the app must first indicate the intention to do so during the creation of the Vulkan object, by including a `VkExportMetalObjectCreateInfoEXT` structure in the `pNext` chain of the `VkInstanceCreateInfo`, `VkMemoryAllocateInfo`, `VkImageCreateInfo`, `VkImageViewCreateInfo`, `VkBufferViewCreateInfo`, `VkSemaphoreCreateInfo`, or `VkEventCreateInfo`, in the corresponding Vulkan object creation command. [source,c] ---- typedef struct VkExportMetalObjectCreateInfoEXT { VkStructureType sType; const void* pNext; VkExportMetalObjectTypeFlagBitsEXT exportObjectType; } VkExportMetalObjectCreateInfoEXT; typedef enum VkExportMetalObjectTypeFlagBitsEXT { VK_EXPORT_METAL_OBJECT_TYPE_METAL_DEVICE_BIT_EXT = 0x00000001, VK_EXPORT_METAL_OBJECT_TYPE_METAL_COMMAND_QUEUE_BIT_EXT = 0x00000002, VK_EXPORT_METAL_OBJECT_TYPE_METAL_BUFFER_BIT_EXT = 0x00000004, VK_EXPORT_METAL_OBJECT_TYPE_METAL_TEXTURE_BIT_EXT = 0x00000008, VK_EXPORT_METAL_OBJECT_TYPE_METAL_IOSURFACE_BIT_EXT = 0x00000010, VK_EXPORT_METAL_OBJECT_TYPE_METAL_SHARED_EVENT_BIT_EXT = 0x00000020, VK_EXPORT_METAL_OBJECT_TYPE_FLAG_BITS_MAX_ENUM_EXT = 0x7FFFFFFF } VkExportMetalObjectTypeFlagBitsEXT; typedef VkFlags VkExportMetalObjectTypeFlagsEXT; ---- === Importing Metal Objects A `VkDeviceMemory` object can be created on an existing `MTLBuffer` object, by including the `MTLBuffer` object in a `VkImportMetalBufferInfoEXT` structure in the `pNext` chain of the `VkMemoryAllocateInfo` structure in the `vkAllocateMemory` command. A `VkImage` object can be created on an existing `IOSurface` object, or one or more existing Metal `MTLTexture` objects, by including those Metal objects in `VkImportMetalIOSurfaceInfoEXT` or `VkImportMetalTextureInfoEXT` structures in the `pNext` chain of the `VkImageCreateInfo` structure in the `vkCreateImage` command. A `VkSemaphore` or `VkEvent` object can be created on an existing `MTLSharedEvent` object, by including the `MTLSharedEvent` object in a `VkImportMetalSharedEventInfoEXT` structure in the `pNext` chain of the `VkSemaphoreCreateInfo` or `VkEventCreateInfo` structure in a `vkCreateSemaphore` or `vkCreateEvent` command, respectively [source,c] ---- typedef struct VkImportMetalBufferInfoEXT { VkStructureType sType; const void* pNext; MTLBuffer_id mtlBuffer; } VkImportMetalBufferInfoEXT; typedef struct VkImportMetalTextureInfoEXT { VkStructureType sType; const void* pNext; VkImageAspectFlags aspectMask; MTLTexture_id mtlTexture; } VkImportMetalTextureInfoEXT; typedef struct VkImportMetalIOSurfaceInfoEXT { VkStructureType sType; const void* pNext; IOSurfaceRef ioSurface; } VkImportMetalIOSurfaceInfoEXT; typedef struct VkImportMetalSharedEventInfoEXT { VkStructureType sType; const void* pNext; MTLSharedEvent_id mtlSharedEvent; } VkImportMetalSharedEventInfoEXT; ---- == Issues === RESOLVED: Should this extension be built by adding Metal platform variations to the `VK_KHR_external_memory` family of extensions? No. While the `VK_KHR_external_memory` family of extensions is suitable for resource objects such as images and buffers, the intent of the `VK_EXT_metal_objects` is to provide consistent access to a wide variety of Metal objects, including, but not limited to resource objects. === RESOLVED: Should this extension be split into two, one for resources and one for other objects? No. Given the expectation that this extension will be limited to layered implementations on top of Metal running only on Apple platforms, the preference was for a single simple extension that served the needs of all expected use cases for apps on those platforms. Furthermore, the existence of this `VK_EXT_metal_objects` extension does not prevent additional resource-only extensions compatible with `VK_KHR_external_memory` from being introduced for those specific use cases, and with the expectation that any resulting functionality overlap will not cause any issues. === RESOLVED: Should the app be required to indicate its intention to export Metal objects when the corresponding Vulkan object is created? Yes. To improve implementation flexibility and integrity, requiring the app to indicate the intention to export provides the implementation with the flexibility to optimize internally if it knows certain Metal objects will or will not be required to be available. === RESOLVED: Should a `MTLBuffer` be imported and exported through a `VkBuffer` or a `VkDeviceMemory`? While `MTLBuffer` spans functionality covered by both `VkBuffer` and `VkDeviceMemory`, it was felt that `MTLBuffer` maps closest to `VkDeviceMemory`, with a `VkBuffer` essentially being a wrapper object around a segment of that `VkDeviceMemory/MTLBuffer`.