Home
last modified time | relevance | path

Searched refs:IsTokenSeparator (Results 1 – 1 of 1) sorted by relevance

/external/libtextclassifier/lang_id/
Dcustom-tokenizer.cc33 inline bool IsTokenSeparator(int num_bytes, const char *curr) { in IsTokenSeparator() function
119 while (IsTokenSeparator(num_bytes, curr)) { in Tokenize()
147 if (IsTokenSeparator(num_bytes, curr)) { in Tokenize()