python中的token bytes_Python token.type方法代碼示例

# 需要導入模塊: import token [as 別名]

# 或者: from token import type [as 別名]

def tokenize(readline):

"""

The tokenize() generator requires one argument, readline, which

must be a callable object which provides the same interface as the

readline() method of built-in file objects. Each call to the function

should return one line of input as bytes. Alternatively, readline

can be a callable function terminating with StopIteration:

readline = open(myfile, 'rb').__next__ # Example of alternate readline

The generator produces 5-tuples with these members: the token type; the

token string; a 2-tuple (srow, scol) of ints specifying the row and

column where the token begins in the source; a 2-tuple (erow, ecol) of

ints specifying the row and column where the token ends in the source;

and the line on which the token was found. The line passed is the

logical line; continuation lines are included.

The first token sequence will always be an ENCODING token

which tells you which encoding was used to decode the bytes stream.

"""

# This import is here to avoid problems when the itertools module is not

# built yet and tokenize is imported.

from itertools import chain, repeat

encoding, consumed = detect_encoding(readline)

rl_gen = iter(readline, b"")

empty = repeat(b"")

return _tokenize(chain(consumed, rl_gen, empty).__next__, encoding)