Lines Matching full:encoding

5 determining source file encoding.
19 operators.  Additionally, all token lists start with an ENCODING token
20 which tells you which encoding was used to decode the bytes stream.
172         self.encoding = None
196             if tok_type == ENCODING:
197                 self.encoding = token
232             if toknum == ENCODING:
233                 self.encoding = tokval
263     It returns a bytes object, encoded using the ENCODING
283     if ut.encoding is not None:
284         out = out.encode(ut.encoding)
301     The detect_encoding() function is used to detect the encoding that should
305     It will call readline a maximum of twice, and return the encoding used
308     It detects the encoding from the presence of a utf-8 bom or an encoding
310     but disagree, a SyntaxError will be raised.  If the encoding cookie is an
314     If no encoding is specified, then the default of 'utf-8' will be returned.
321     encoding = None
331             # Decode as UTF-8. Either the line is an encoding declaration,
333             # per default encoding.
336             msg = "invalid or missing encoding declaration"
344         encoding = _get_normal_name(match.group(1))
346             codec = lookup(encoding)
350                 msg = "unknown encoding: " + encoding
352                 msg = "unknown encoding for {!r}: {}".format(filename,
353                         encoding)
357             if encoding != 'utf-8':
360                     msg = 'encoding problem: utf-8'
362                     msg = 'encoding problem for {!r}: utf-8'.format(filename)
364             encoding += '-sig'
365         return encoding
375     encoding = find_cookie(first)
376     if encoding:
377         return encoding, [first]
385     encoding = find_cookie(second)
386     if encoding:
387         return encoding, [first, second]
393     """Open a file in read only mode using the encoding detected by
398         encoding, lines = detect_encoding(buffer.readline)
400         text = TextIOWrapper(buffer, encoding, line_buffering=True)
424     The first token sequence will always be an ENCODING token
425     which tells you which encoding was used to decode the bytes stream.
427     encoding, consumed = detect_encoding(readline)
430     return _tokenize(rl_gen.__next__, encoding)
433 def _tokenize(readline, encoding):  argument
440     if encoding is not None:
441         if encoding == "utf-8-sig":
443             encoding = "utf-8"
444         yield TokenInfo(ENCODING, encoding, (0, 0), (0, 0), '')
458         if encoding is not None:
459             line = line.decode(encoding)