Searched defs:BasicTokenizer (Results 1 – 2 of 2) sorted by relevance
1078 class BasicTokenizer(TextTensorOperation): class
69 BasicTokenizer::BasicTokenizer(bool lower_case, bool keep_whitespace, const NormalizeForm normalize… in BasicTokenizer() function in mindspore::dataset::text::BasicTokenizer