Searched refs:input_tokenize_results (Results 1 – 1 of 1) sorted by relevance
78 TokenizerResult input_tokenize_results; in Preprocess() local79 input_tokenize_results = tokenizer_->Tokenize(processed_input); in Preprocess()83 absl::MakeSpan(input_tokenize_results.subwords.data(), in Preprocess()84 input_tokenize_results.subwords.data() + in Preprocess()86 input_tokenize_results.subwords.size())); in Preprocess()