Searched defs:BasicTokenizer (Results 1 – 3 of 3) sorted by relevance
165 class BasicTokenizer: class
567 class BasicTokenizer(TextTensorOperation): class
59 BasicTokenizer::BasicTokenizer(bool lower_case, bool keep_whitespace, const NormalizeForm normalize… in BasicTokenizer() function in mindspore::dataset::text::BasicTokenizer