Searched refs:token_start_codepoint (Results 1 – 1 of 1) sorted by relevance
72 int token_start_codepoint = 0; in TokenizeWithFilter() local82 Token{token_text, token_start_codepoint, codepoint_idx}); in TokenizeWithFilter()90 token_start_codepoint = codepoint_idx + 1; in TokenizeWithFilter()98 tokens.push_back(Token{token_text, token_start_codepoint, codepoint_idx}); in TokenizeWithFilter()