Searched defs:max_bytes_per_token (Results 1 – 9 of 9) sorted by relevance
| /third_party/mindspore/tests/ut/python/dataset/ |
| D | test_text_wordpiece_tokenizer.py | 100 vocab_list, unknown_token='[UNK]', max_bytes_per_token=100): argument
120 … vocab_list, unknown_token='[UNK]', max_bytes_per_token=100): argument
|
| D | test_text_bert_tokenizer.py | 173 max_bytes_per_token=100, unknown_token='[UNK]', argument
202 max_bytes_per_token=100, unknown_token='[UNK]', argument
|
| /third_party/mindspore/mindspore/ccsrc/minddata/dataset/text/kernels/ |
| D | bert_tokenizer_op.h | 42 … : wordpiece_tokenizer_(vocab, suffix_indicator, max_bytes_per_token, unknown_token, with_offsets), in wordpiece_tokenizer_() argument
|
| D | wordpiece_tokenizer_op.cc | 30 … const int &max_bytes_per_token, const std::string &unknown_token, in WordpieceTokenizerOp()
|
| /third_party/mindspore/mindspore/ccsrc/minddata/dataset/include/dataset/ |
| D | text.h | 98 …: BertTokenizer(vocab, StringToChar(suffix_indicator), max_bytes_per_token, StringToChar(unknown_t… in BertTokenizer() argument
537 …: WordpieceTokenizer(vocab, StringToChar(suffix_indicator), max_bytes_per_token, StringToChar(unkn… in WordpieceTokenizer() argument
|
| /third_party/mindspore/mindspore/ccsrc/minddata/dataset/api/python/bindings/dataset/text/kernels/ir/ |
| D | bindings.cc | 48 bool with_offsets) { in __anone8b21dd50302()
238 … int32_t max_bytes_per_token, const std::string &unknown_token, bool with_offsets) { in __anone8b21dd52402()
|
| /third_party/mindspore/mindspore/ccsrc/minddata/dataset/text/ir/kernels/ |
| D | text_ir.cc | 87 … int32_t max_bytes_per_token, const std::string &unknown_token, in BertTokenizerOperation()
476 … int32_t max_bytes_per_token, const std::string &unknown_token, in WordpieceTokenizerOperation()
|
| /third_party/mindspore/mindspore/dataset/text/ |
| D | transforms.py | 513 def __init__(self, vocab, suffix_indicator='##', max_bytes_per_token=100, argument
691 … def __init__(self, vocab, suffix_indicator='##', max_bytes_per_token=100, unknown_token='[UNK]', argument
|
| /third_party/mindspore/mindspore/ccsrc/minddata/dataset/api/ |
| D | text.cc | 94 … int32_t max_bytes_per_token, const std::vector<char> &unknown_token, bool lower_case, in BertTokenizer()
405 … int32_t max_bytes_per_token, const std::vector<char> &unknown_token, in WordpieceTokenizer()
|