Searched refs:FindTokenizationRole (Results 1 – 3 of 3) sorted by relevance
39 TokenizationCodepointRange::Role Tokenizer::FindTokenizationRole( in FindTokenizationRole() function in libtextclassifier::Tokenizer75 TokenizationCodepointRange::Role role = FindTokenizationRole(*it); in Tokenize()
60 TokenizationCodepointRange::Role FindTokenizationRole(int codepoint) const;
34 return FindTokenizationRole(c); in TestFindTokenizationRole()38 TEST(TokenizerTest, FindTokenizationRole) { in TEST() argument