Searched defs:generate_tokens (Results 1 – 2 of 2) sorted by relevance
3649 def generate_tokens(self, text): member in CachedTokenizer3660 generate_tokens = _cached_tokenizer.generate_tokens variable
1363 def generate_tokens(self): member in Checker