Lines Matching +full:use +full:- +full:kernel
1 # Kernel Registration
4 …-overview.md), we lower the operators in the dialect to the _out variants_ of the [core ATen opera…
6 ## Kernel Libraries
7 ### First-party kernel libraries:
9 …kernel library](https://github.com/pytorch/executorch/tree/main/kernels/portable)** is the in-hous…
11 …kernel library](https://github.com/pytorch/executorch/tree/main/kernels/optimized)** specializes o…
13 …kernel library](https://github.com/pytorch/executorch/tree/main/kernels/quantized)** implements op…
15 ### Custom kernel libraries:
17 …kernel library can be viewed as a good example. We have optimized [`add.out`](https://github.com/p…
18 1. combining kernel libraries and define fallback kernels;
19 2. using selective build to minimize the kernel size.
23 ## Operator & Kernel Contract
25 All the kernels mentioned above, whether they are in-house or customized, should comply with the fo…
27 * Match the calling convention derived from operator schema. The kernel registration API will gener…
28 …ct. For tensors with certain dtypes as arguments, the result of a custom kernel needs to match the…
36 * [YAML Entry API](#yaml-entry-api-high-level-architecture)
37 - [for core ATen op with custom kernels](#yaml-entry-api-for-core-aten-op-out-variant)
38 - [for custom ops](#yaml-entry-api-for-custom-ops)
39 - [CMake Macros](#cmake-macros)
41 - [for custom ops](#c-api-for-custom-ops)
42 - [CMake Example](#compile-and-link-the-custom-kernel)
44 If it's not clear which API to use, please see [Best Practices](#custom-ops-api-best-practices).
50 
54 1. the custom kernel library with C++ implementations
56 …also contains information on the dtypes and dim orders supported by the kernel. More details in t…
61 …kernel libraries will be passed to the _kernel resolver_ along with the model op info (see selecti…
63 At static object initialization time, kernels will be registered into the ExecuTorch kernel registr…
65 …use the operator name and argument metadata as a key to lookup for the kernels. For example, with …
72 …kernel information. It consists of `arg_meta` and `kernel_name`, which are bound together to descr…
80 …lemented by the corresponding `kernel_name`. This being `null` means the kernel will be used for a…
85 - op: add.out
87 - arg_meta: null
90 An out variant of a core ATen operator with a default kernel
92 ATen operator with a dtype/dim order specialized kernel (works for `Double` dtype and dim order nee…
94 - op: add.out
100 - arg_meta:
113 …t we need to specify the operator schema as well as a `kernel` section. So instead of `op` we use …
115 - func: allclose.out(Tensor self, Tensor other, float rtol=1e-05, float atol=1e-08, bool equal_nan=…
117 - arg_meta: null
120 The `kernel` section is the same as the one defined in core ATen ops. For operator schema, we are r…
125 ExecuTorch only supports out-style operators, where:
131 * The `out` argument must be keyword-only, which means it needs to follow an argument named `*` lik…
134 …ue. But, to be consistent, functions should always return `out` when the return type is non-`void`.
161 …macros to help users to build their kernel registration library. The macro takes the yaml file des…
164 … variants and also a yaml file for custom ops, generate C++ bindings for kernel registration. It a…
186 - op: add.out
188 - arg_meta: null
196 - op: add.out
198 - arg_meta: null
206 …C++ API only uses C++ macros `EXECUTORCH_LIBRARY` and `WRAP_TO_ATEN` for kernel registration, also…
208 Please refer to [Custom Ops Best Practices](#custom-ops-api-best-practices) on which API to use.
212 #### Prepare custom kernel implementation
217 custom_linear(Tensor weight, Tensor input, Tensor(?) bias) -> Tensor
218 custom_linear.out(Tensor weight, Tensor input, Tensor(?) bias, *, Tensor(a!) out) -> Tensor(a!)
221 Then write your custom kernel according to the schema using ExecuTorch types, along with APIs to re…
226 #include <executorch/runtime/kernel/kernel_includes.h>
232 #### Use a C++ macro to register it into ExecuTorch
241 …s op to show up in PyTorch, but don’t worry we don’t need to rewrite the kernel. Create a separate…
251 // wrap kernel in custom_linear.cpp into ATen kernel
257 m.def("custom_linear(Tensor weight, Tensor input, Tensor(?) bias) -> Tensor", custom_linear);
258 …m.def("custom_linear.out(Tensor weight, Tensor input, Tensor(?) bias, *, Tensor(a!) out) -> Tensor…
262 #### Compile and link the custom kernel
284 # Link this library with --whole-archive !! IMPORTANT !! this is to avoid the operators being strip…
298 # Now we have access to the custom op, backed by kernel implemented in custom_linear.cpp.
304 Given that we have 2 kernel registration APIs for custom ops, which API should we use? Here are som…
307 - Pros:
311 - Cons:
316 - Pros:
319 …- It shows what ops are being registered and what kernels are bound to these ops, for an applicati…
320 - Cons:
324 …use the C++ API since it's low-cost to use and flexible to change. Once the application moves to p…