Searched defs:normalize_form (Results 1 – 7 of 7) sorted by relevance
| /third_party/mindspore/mindspore/ccsrc/minddata/dataset/text/kernels/ |
| D | normalize_utf8_op.h | 31 …explicit NormalizeUTF8Op(NormalizeForm normalize_form = kDefNormalizeForm) : normalize_form_(norma… in normalize_form_() argument
|
| /third_party/mindspore/mindspore/ccsrc/minddata/dataset/api/python/bindings/dataset/text/kernels/ir/ |
| D | bindings.cc | 47 … bool keep_whitespace, const NormalizeForm normalize_form, bool preserve_unused_token, in __anonf0f064540302()
48 bool with_offsets) { in __anonf0f064540302()
71 .def(py::init([](NormalizeForm normalize_form) { in __anonf0f064540702()
|
| /third_party/mindspore/mindspore/ccsrc/minddata/dataset/text/ir/kernels/ |
| D | text_ir.cc | 60 … const NormalizeForm normalize_form, bool preserve_unused_token, in BasicTokenizerOperation()
89 … const NormalizeForm normalize_form, bool preserve_unused_token, in BertTokenizerOperation()
288 NormalizeUTF8Operation::NormalizeUTF8Operation(NormalizeForm normalize_form) : normalize_form_(norm… in NormalizeUTF8Operation()
|
| /third_party/mindspore/tests/ut/python/dataset/ |
| D | test_text_tokenizer.py | 242 def normalize(normalize_form): argument
|
| /third_party/mindspore/mindspore/ccsrc/minddata/dataset/include/dataset/ |
| D | text.h | 99 … lower_case, keep_whitespace, normalize_form, preserve_unused_token, with_offsets) {} in BertTokenizer() argument
|
| /third_party/mindspore/mindspore/ccsrc/minddata/dataset/api/ |
| D | text.cc | 59 …okenizer::BasicTokenizer(bool lower_case, bool keep_whitespace, const NormalizeForm normalize_form, in BasicTokenizer()
95 … bool keep_whitespace, const NormalizeForm normalize_form, bool preserve_unused_token, in BertTokenizer()
271 NormalizeUTF8::NormalizeUTF8(NormalizeForm normalize_form) : data_(std::make_shared<Data>(normalize… in NormalizeUTF8()
|
| /third_party/mindspore/mindspore/dataset/text/ |
| D | transforms.py | 755 def __init__(self, normalize_form=NormalizeForm.NFKC): argument
|