Lines Matching +full:detect +full:- +full:newline

7 text into Python tokens.  It accepts a readline-like method which is called
9 5-tuples with these members:
13     the starting (row, column) indices of the token (a 2-tuple of ints)
14     the ending (row, column) indices of the token (a 2-tuple of ints)
28 __author__ = 'Ka-Ping Yee <ping@lfw.org>'
62 Hexnumber = r'0[xX]_?[\da-fA-F]+(?:_[\da-fA-F]+)*[lL]?'
63 Octnumber = r'0[oO]?_?[0-7]+(?:_[0-7]+)*[lL]?'
64 Decnumber = group(r'[1-9]\d*(?:_\d+)*[lL]?', '0[lL]?')
66 Exponent = r'[eE][-+]?\d+(?:_\d+)*'
83 # Single-line ' or " string.
87 # Because of leftmost-then-longest match semantics, be sure to put the
91                  r"//=?", r"->",
92                  r"[+\-*/%&@|^=<>]=?",
145     print("%d,%d-%d,%d:\t%s\t%s" % \
154     the same interface as the readline() method of built-in file objects.
181         col_offset = col - self.prev_col
194             if tok_type in (NEWLINE, NL):
206         if toknum in (NEWLINE, NL):
220             elif toknum in (NEWLINE, NL):
223                 toks_append(indents[-1])
227 cookie_re = re.compile(r'^[ \t\f]*#.*?coding[:=][ \t]*([-\w.]+)', re.ASCII)
233     enc = orig_enc[:12].lower().replace("_", "-")
234     if enc == "utf-8" or enc.startswith("utf-8-"):
235         return "utf-8"
236     if enc in ("latin-1", "iso-8859-1", "iso-latin-1") or \
237        enc.startswith(("latin-1-", "iso-8859-1-", "iso-latin-1-")):
238         return "iso-8859-1"
243     The detect_encoding() function is used to detect the encoding that should
251     It detects the encoding from the presence of a utf-8 bom or an encoding
252     cookie as specified in pep-0263. If both a bom and a cookie are present, but
254     charset, raise a SyntaxError.  Note that if a utf-8 bom is found,
255     'utf-8-sig' is returned.
257     If no encoding is specified, then the default of 'utf-8' will be returned.
261     default = 'utf-8'
284             if codec.name != 'utf-8':
286                 raise SyntaxError('encoding problem: utf-8')
287             encoding += '-sig'
294         default = 'utf-8-sig'
321     Round-trip invariant for full input:
324     Round-trip invariant for limited input:
339     readline() method of built-in file objects. Each call to the function
344     The generator produces 5-tuples with these members: the token type; the
345     token string; a 2-tuple (srow, scol) of ints specifying the row and
346     column where the token begins in the source; a 2-tuple (erow, ecol) of
372                 raise TokenError("EOF in multi-line string", strstart)
380             elif needcont and line[-2:] != '\\\n' and line[-3:] != '\\\r\n':
419             if column > indents[-1]:           # count indents or dedents
422             while column < indents[-1]:
427                 indents = indents[:-1]
429                 if async_def and async_def_indent >= indents[-1]:
436             if async_def and async_def_nl and async_def_indent >= indents[-1]:
443                 raise TokenError("EOF in multi-line statement", (lnum, 0))
457                     newline = NEWLINE
459                         newline = NL
465                     yield (newline, token, spos, epos, line)
491                     if token[-1] == '\n':                  # continued string
522                                 async_def_indent = indents[-1]
543                     elif initial in ')]}': parenlev = parenlev - 1