Searched defs:generate_tokens (Results 1 – 1 of 1) sorted by relevance
144 def generate_tokens(self, text): member in CachedTokenizer153 generate_tokens = CachedTokenizer().generate_tokens variable