Searched defs:byte_end (Results 1 – 1 of 1) sorted by relevance
| /external/libtextclassifier/native/utils/ |
| D | wordpiece_tokenizer.cc | 28 LookupStatus Lookup(int byte_start, int byte_end, const absl::string_view token, in Lookup()
52 int* byte_end, bool* found_match, bool* match_is_unknown_character) { in LongestMatchStartingAt()
112 void AddWord(const absl::string_view token, int byte_start, int byte_end, in AddWord()
132 int byte_end, const std::string& suffix_indicator, in AddUnknownCharacter()
171 int byte_end; in TokenizeL2RGreedy() local
|