Searched defs:unicode_text (Results 1 – 2 of 2) sorted by relevance
160 void Tokenizer::TokenizeSubstring(const UnicodeText& unicode_text, in TokenizeSubstring()185 void Tokenizer::InternalRetokenize(const UnicodeText& unicode_text, in InternalRetokenize()
1711 PyObject *unicode_text = PyUnicode_DecodeUTF8(text, len, "replace"); in dec_utf8() local