Searched defs:max_bytes_per_token (Results 1 – 9 of 9) sorted by relevance
| /third_party/mindspore/mindspore-src/source/tests/ut/python/dataset/ |
| D | test_text_wordpiece_tokenizer.py | 100 vocab_list, unknown_token='[UNK]', max_bytes_per_token=100): argument
120 … vocab_list, unknown_token='[UNK]', max_bytes_per_token=100): argument
|
| D | test_text_bert_tokenizer.py | 173 max_bytes_per_token=100, unknown_token='[UNK]', argument
202 max_bytes_per_token=100, unknown_token='[UNK]', argument
|
| /third_party/mindspore/mindspore-src/source/mindspore/ccsrc/minddata/dataset/text/kernels/ |
| D | bert_tokenizer_op.h | 42 … : wordpiece_tokenizer_(vocab, suffix_indicator, max_bytes_per_token, unknown_token, with_offsets), in wordpiece_tokenizer_() argument
|
| D | wordpiece_tokenizer_op.cc | 30 … const int &max_bytes_per_token, const std::string &unknown_token, in WordpieceTokenizerOp()
|
| /third_party/mindspore/mindspore-src/source/mindspore/ccsrc/minddata/dataset/include/dataset/ |
| D | text.h | 345 …: BertTokenizer(vocab, StringToChar(suffix_indicator), max_bytes_per_token, StringToChar(unknown_t… in BertTokenizer() argument
1010 …: WordpieceTokenizer(vocab, StringToChar(suffix_indicator), max_bytes_per_token, StringToChar(unkn… in WordpieceTokenizer() argument
|
| /third_party/mindspore/mindspore-src/source/mindspore/ccsrc/minddata/dataset/api/python/bindings/dataset/text/kernels/ir/ |
| D | bindings.cc | 58 bool with_offsets) { in __anon702ab2a90502()
281 … int32_t max_bytes_per_token, const std::string &unknown_token, bool with_offsets) { in __anon702ab2a92c02()
|
| /third_party/mindspore/mindspore-src/source/mindspore/python/mindspore/dataset/text/ |
| D | transforms.py | 1054 def __init__(self, vocab, suffix_indicator='##', max_bytes_per_token=100, unknown_token='[UNK]', argument
1271 … def __init__(self, vocab, suffix_indicator='##', max_bytes_per_token=100, unknown_token='[UNK]', argument
|
| /third_party/mindspore/mindspore-src/source/mindspore/ccsrc/minddata/dataset/text/ir/kernels/ |
| D | text_ir.cc | 121 … int32_t max_bytes_per_token, const std::string &unknown_token, in BertTokenizerOperation()
649 … int32_t max_bytes_per_token, const std::string &unknown_token, in WordpieceTokenizerOperation()
|
| /third_party/mindspore/mindspore-src/source/mindspore/ccsrc/minddata/dataset/api/ |
| D | text.cc | 104 … int32_t max_bytes_per_token, const std::vector<char> &unknown_token, bool lower_case, in BertTokenizer()
448 … int32_t max_bytes_per_token, const std::vector<char> &unknown_token, in WordpieceTokenizer()
|