Lines Matching refs:tokenize

1 :mod:`tokenize` --- Tokenizer for Python source
4 .. module:: tokenize
9 **Source code:** :source:`Lib/tokenize.py`
13 The :mod:`tokenize` module provides a lexical scanner for Python source code,
22 :func:`tokenize.generate_tokens` for the character sequence that identifies a
48 .. function:: tokenize(readline[, tokeneater])
50    The :func:`.tokenize` function accepts two parameters: one representing the input
51    stream, and one providing an output mechanism for :func:`.tokenize`.
67 :mod:`tokenize`, as are two additional token type values that might be passed to
68 the *tokeneater* function by :func:`.tokenize`:
83 for creating tools that tokenize a script, modify the token stream, and write
94    guaranteed to tokenize back to match the input so that the conversion is
137        g = generate_tokens(StringIO(s).readline)   # tokenize the string