Searched refs:token_utf8 (Results 1 – 1 of 1) sorted by relevance
683 std::string token_utf8; in ICUTokenize() local684 token.toUTF8String(token_utf8); in ICUTokenize()694 result->push_back(Token(token_utf8, last_unicode_index, unicode_index)); in ICUTokenize()