Searched defs:Tokenize (Results 1 – 10 of 10) sorted by relevance
| /third_party/mindspore/mindspore/ccsrc/minddata/dataset/text/kernels/ |
| D | tokenizer_op.h | 37 …virtual Status Tokenize(std::string_view str, std::vector<std::string> *splits, std::vector<uint32… in Tokenize() function
|
| D | unicode_char_tokenizer_op.cc | 29 Status UnicodeCharTokenizerOp::Tokenize(std::string_view str, std::vector<std::string> *splits, in Tokenize() function in mindspore::dataset::UnicodeCharTokenizerOp
|
| D | whitespace_tokenizer_op.cc | 30 Status WhitespaceTokenizerOp::Tokenize(std::string_view str, std::vector<std::string> *splits, in Tokenize() function in mindspore::dataset::WhitespaceTokenizerOp
|
| D | unicode_script_tokenizer_op.cc | 36 Status UnicodeScriptTokenizerOp::Tokenize(std::string_view str, std::vector<std::string> *splits, in Tokenize() function in mindspore::dataset::UnicodeScriptTokenizerOp
|
| D | jieba_tokenizer_op.cc | 32 Status JiebaTokenizerOp::Tokenize(std::string_view sentence_v, std::vector<std::string> *words, in Tokenize() function in mindspore::dataset::JiebaTokenizerOp
|
| D | regex_tokenizer_op.cc | 110 Status RegexTokenizerOp::Tokenize(std::string_view str, std::vector<std::string> *splits, in Tokenize() function in mindspore::dataset::RegexTokenizerOp
|
| /third_party/flutter/skia/third_party/externals/wuffs/lang/token/ |
| D | token.go | 116 func Tokenize(m *Map, filename string, src []byte) (tokens []Token, comments []string, retErr error… func
|
| /third_party/mindspore/mindspore/lite/src/common/ |
| D | utils.cc | 102 std::vector<std::string> Tokenize(const std::string &src, const std::string &delimiters, in Tokenize() function
|
| /third_party/gn/src/gn/ |
| D | tokenizer.cc | 82 std::vector<Token> Tokenizer::Tokenize( in Tokenize() function in Tokenizer
|
| /third_party/node/tools/ |
| D | test.py | 1070 def Tokenize(self): member in Tokenizer
|