Searched defs:normalize_form (Results 1 – 7 of 7) sorted by relevance
| /third_party/mindspore/mindspore-src/source/mindspore/ccsrc/minddata/dataset/text/kernels/ |
| D | normalize_utf8_op.h | 31 …explicit NormalizeUTF8Op(NormalizeForm normalize_form = kDefNormalizeForm) : normalize_form_(norma… in normalize_form_() argument
|
| /third_party/mindspore/mindspore-src/source/mindspore/ccsrc/minddata/dataset/api/python/bindings/dataset/text/kernels/ir/ |
| D | bindings.cc | 57 … bool keep_whitespace, const NormalizeForm normalize_form, bool preserve_unused_token, in __anon2bada04b0502()
58 bool with_offsets) { in __anon2bada04b0502()
92 .def(py::init([](NormalizeForm normalize_form) { in __anon2bada04b0b02()
|
| /third_party/mindspore/mindspore-src/source/tests/ut/python/dataset/ |
| D | test_text_tokenizer.py | 258 def normalize(normalize_form): argument
|
| /third_party/mindspore/mindspore-src/source/mindspore/ccsrc/minddata/dataset/text/ir/kernels/ |
| D | text_ir.cc | 95 … const NormalizeForm normalize_form, bool preserve_unused_token, in BasicTokenizerOperation()
123 … const NormalizeForm normalize_form, bool preserve_unused_token, in BertTokenizerOperation()
427 NormalizeUTF8Operation::NormalizeUTF8Operation(NormalizeForm normalize_form) : normalize_form_(norm… in NormalizeUTF8Operation()
|
| /third_party/mindspore/mindspore-src/source/mindspore/ccsrc/minddata/dataset/include/dataset/ |
| D | text.h | 346 … lower_case, keep_whitespace, normalize_form, preserve_unused_token, with_offsets) {} in BertTokenizer() argument
|
| /third_party/mindspore/mindspore-src/source/mindspore/ccsrc/minddata/dataset/api/ |
| D | text.cc | 69 …okenizer::BasicTokenizer(bool lower_case, bool keep_whitespace, const NormalizeForm normalize_form, in BasicTokenizer()
105 … bool keep_whitespace, const NormalizeForm normalize_form, bool preserve_unused_token, in BertTokenizer()
287 NormalizeUTF8::NormalizeUTF8(NormalizeForm normalize_form) : data_(std::make_shared<Data>(normalize… in NormalizeUTF8()
|
| /third_party/mindspore/mindspore-src/source/mindspore/python/mindspore/dataset/text/ |
| D | transforms.py | 1417 def __init__(self, normalize_form=NormalizeForm.NFKC): argument
|