Home
last modified time | relevance | path

Searched refs:AppendCodepointToToken (Results 1 – 1 of 1) sorted by relevance

/external/libtextclassifier/native/utils/
Dtokenizer.cc131 void AppendCodepointToToken(UnicodeText::const_iterator it, Token* token) { in AppendCodepointToToken() function
159 AppendCodepointToToken(it, &new_token); in InternalTokenize()
302 AppendCodepointToToken(it, &new_token); in NumberTokenize()
312 AppendCodepointToToken(it, &result->back()); in NumberTokenize()
332 AppendCodepointToToken(it, &new_token); in NumberTokenize()