1:tocdepth: 2 2 3.. _module-pw_tokenizer-api: 4 5========================== 6pw_tokenizer API reference 7========================== 8.. pigweed-module-subpage:: 9 :name: pw_tokenizer 10 11.. _module-pw_tokenizer-api-configuration: 12 13------------- 14Configuration 15------------- 16.. tab-set:: 17 18 .. tab-item:: C++ / C 19 :sync: cpp 20 21 .. doxygenfile:: pw_tokenizer/config.h 22 :sections: define 23 24------------ 25Tokenization 26------------ 27.. tab-set:: 28 29 .. tab-item:: C++ / C 30 :sync: cpp 31 32 .. doxygenfunction:: pw::tokenizer::EncodeArgs 33 .. doxygenclass:: pw::tokenizer::EncodedMessage 34 :members: 35 .. doxygenfunction:: pw::tokenizer::MinEncodingBufferSizeBytes 36 .. doxygendefine:: PW_TOKEN_FMT 37 .. doxygendefine:: PW_TOKENIZE_FORMAT_STRING 38 .. doxygendefine:: PW_TOKENIZE_FORMAT_STRING_ANY_ARG_COUNT 39 .. doxygendefine:: PW_TOKENIZE_STRING 40 .. doxygendefine:: PW_TOKENIZE_STRING_DOMAIN 41 .. doxygendefine:: PW_TOKENIZE_STRING_DOMAIN_EXPR 42 .. doxygendefine:: PW_TOKENIZE_STRING_EXPR 43 .. doxygendefine:: PW_TOKENIZE_STRING_MASK 44 .. doxygendefine:: PW_TOKENIZE_STRING_MASK_EXPR 45 .. doxygendefine:: PW_TOKENIZE_TO_BUFFER 46 .. doxygendefine:: PW_TOKENIZE_TO_BUFFER_DOMAIN 47 .. doxygendefine:: PW_TOKENIZE_TO_BUFFER_MASK 48 .. doxygendefine:: PW_TOKENIZER_REPLACE_FORMAT_STRING 49 .. doxygendefine:: PW_TOKENIZER_ARG_TYPES 50 .. doxygenfunction:: pw_tokenizer_EncodeArgs 51 .. doxygenfunction:: pw_tokenizer_EncodeInt 52 .. doxygenfunction:: pw_tokenizer_EncodeInt64 53 .. doxygentypedef:: pw_tokenizer_Token 54 55 .. tab-item:: Python 56 :sync: py 57 58 .. autofunction:: pw_tokenizer.encode.encode_token_and_args 59 .. autofunction:: pw_tokenizer.tokens.pw_tokenizer_65599_hash 60 61 .. tab-item:: Rust 62 :sync: rs 63 64 See `Crate pw_tokenizer </rustdoc/pw_tokenizer/>`_. 65 66.. _module-pw_tokenizer-api-token-databases: 67 68--------------- 69Token databases 70--------------- 71.. tab-set:: 72 73 .. tab-item:: C++ / C 74 :sync: cpp 75 76 .. doxygenclass:: pw::tokenizer::TokenDatabase 77 :members: 78 79.. _module-pw_tokenizer-api-detokenization: 80 81-------------- 82Detokenization 83-------------- 84.. tab-set:: 85 86 .. tab-item:: C++ / C 87 :sync: cpp 88 89 .. doxygengroup:: pw_tokenizer_detokenize 90 :content-only: 91 :members: 92 93 .. tab-item:: Python 94 :sync: py 95 96 .. automodule:: pw_tokenizer.detokenize 97 :members: 98 99 .. automodule:: pw_tokenizer.proto 100 :members: 101