Home
last modified time | relevance | path

Searched defs:unknown_token (Results 1 – 5 of 5) sorted by relevance

/external/libtextclassifier/native/utils/
Dwordpiece_tokenizer.cc93 const std::string& unknown_token, in NoTokenFound()
134 const std::string& unknown_token, in AddUnknownCharacter()
162 bool use_unknown_token, const std::string& unknown_token, in TokenizeL2RGreedy()
212 bool use_unknown_token, const std::string& unknown_token, in WordpieceTokenize()
238 const std::string& unknown_token, const WordpieceVocab* vocab_map, in WordpieceTokenize()
Dbert_tokenizer.h62 std::string unknown_token = kDefaultUnknownToken; member
/external/tflite-support/tensorflow_lite_support/cc/text/tokenizers/
Dbert_tokenizer_jni.cc48 std::string unknown_token(raw_unknown_token); in Java_org_tensorflow_lite_support_text_tokenizers_BertTokenizer_nativeLoadResource() local
Dregex_tokenizer.cc118 bool RegexTokenizer::GetUnknownToken(int* unknown_token) { in GetUnknownToken()
Dbert_tokenizer.h64 std::string unknown_token = kDefaultUnknownToken; member