1:tocdepth: 2 2 3.. _module-pw_tokenizer-api: 4 5========================== 6pw_tokenizer API reference 7========================== 8.. pigweed-module-subpage:: 9 :name: pw_tokenizer 10 11.. _module-pw_tokenizer-api-configuration: 12 13------------- 14Configuration 15------------- 16.. tab-set:: 17 18 .. tab-item:: C++ / C 19 :sync: cpp 20 21 .. doxygenfile:: pw_tokenizer/config.h 22 :sections: define 23 24------------ 25Tokenization 26------------ 27.. tab-set:: 28 29 .. tab-item:: C++ / C 30 :sync: cpp 31 32 .. doxygenfunction:: pw::tokenizer::EncodeArgs 33 .. doxygenclass:: pw::tokenizer::EncodedMessage 34 :members: 35 .. doxygenfunction:: pw::tokenizer::MinEncodingBufferSizeBytes 36 .. doxygenfunction:: pw::tokenizer::EnumToToken 37 .. doxygenfunction:: pw::tokenizer::EnumToString 38 .. doxygendefine:: PW_TOKEN_FMT 39 .. doxygendefine:: PW_NESTED_TOKEN_FMT 40 .. doxygendefine:: PW_TOKENIZE_ENUM 41 .. doxygendefine:: PW_TOKENIZE_ENUM_CUSTOM 42 .. doxygendefine:: PW_TOKENIZE_FORMAT_STRING 43 .. doxygendefine:: PW_TOKENIZE_FORMAT_STRING_ANY_ARG_COUNT 44 .. doxygendefine:: PW_TOKENIZE_STRING 45 .. doxygendefine:: PW_TOKENIZE_STRING_DOMAIN 46 .. doxygendefine:: PW_TOKENIZE_STRING_DOMAIN_EXPR 47 .. doxygendefine:: PW_TOKENIZE_STRING_EXPR 48 .. doxygendefine:: PW_TOKENIZE_STRING_MASK 49 .. doxygendefine:: PW_TOKENIZE_STRING_MASK_EXPR 50 .. doxygendefine:: PW_TOKENIZE_TO_BUFFER 51 .. doxygendefine:: PW_TOKENIZE_TO_BUFFER_DOMAIN 52 .. doxygendefine:: PW_TOKENIZE_TO_BUFFER_MASK 53 .. doxygendefine:: PW_TOKENIZER_REPLACE_FORMAT_STRING 54 .. doxygendefine:: PW_TOKENIZER_ARG_TYPES 55 .. doxygendefine:: PW_TOKENIZER_DEFINE_TOKEN 56 .. doxygenfunction:: pw_tokenizer_EncodeArgs 57 .. doxygenfunction:: pw_tokenizer_EncodeInt 58 .. doxygenfunction:: pw_tokenizer_EncodeInt64 59 .. doxygentypedef:: pw_tokenizer_Token 60 61 .. tab-item:: Python 62 :sync: py 63 64 .. autofunction:: pw_tokenizer.encode.encode_token_and_args 65 .. autofunction:: pw_tokenizer.tokens.pw_tokenizer_65599_hash 66 67 .. tab-item:: Rust 68 :sync: rs 69 70 See `Crate pw_tokenizer </rustdoc/pw_tokenizer/>`_. 71 72.. _module-pw_tokenizer-api-token-databases: 73 74--------------- 75Token databases 76--------------- 77.. tab-set:: 78 79 .. tab-item:: C++ / C 80 :sync: cpp 81 82 .. doxygenclass:: pw::tokenizer::TokenDatabase 83 :members: 84 85.. _module-pw_tokenizer-api-detokenization: 86 87-------------- 88Detokenization 89-------------- 90.. tab-set:: 91 92 .. tab-item:: C++ / C 93 :sync: cpp 94 95 .. doxygengroup:: pw_tokenizer_detokenize 96 :content-only: 97 :members: 98 99 .. tab-item:: Python 100 :sync: py 101 102 .. automodule:: pw_tokenizer.detokenize 103 :members: 104 105 .. automodule:: pw_tokenizer.proto 106 :members: 107