Searched refs:split_unknown_characters (Results 1 – 2 of 2) sorted by relevance
| /external/libtextclassifier/native/utils/ |
| D | wordpiece_tokenizer.cc | 51 bool split_unknown_characters, const WordpieceVocab* vocab_map, in LongestMatchStartingAt() argument
80 if (i == 0 && split_unknown_characters) { in LongestMatchStartingAt()
163 bool split_unknown_characters, const WordpieceVocab* vocab_map, in TokenizeL2RGreedy() argument
176 split_unknown_characters, vocab_map, &byte_end, &found_subword, in TokenizeL2RGreedy()
213 bool split_unknown_characters, const WordpieceVocab* vocab_map, in WordpieceTokenize() argument
230 split_unknown_characters, vocab_map, subwords, in WordpieceTokenize()
|
| D | wordpiece_tokenizer.h | 48 bool split_unknown_characters, const WordpieceVocab* vocab_map,
|