Searched defs:SentencePieceTokenizer (Results 1 – 1 of 1) sorted by relevance
35 explicit SentencePieceTokenizer(const std::string& path_to_model) { in SentencePieceTokenizer() function39 explicit SentencePieceTokenizer(const char* spmodel_buffer_data, in SentencePieceTokenizer() function