python tokenize怎么用_tokenize -- 对 Python 代码使用的标记解析器 — Python 3.7.9 文档...

对输入进行解析标记¶

主要的入口是一个 generator:

tokenize.tokenize(readline)¶

生成器 tokenize() 需要一个 readline 参数，这个参数必须是一个可调用对象，且能提供与文件对象的 io.IOBase.readline() 方法相同的接口。每次调用这个函数都要 返回字节类型输入的一行数据。

The generator produces 5-tuples with these members: the token type; the

token string; a 2-tuple (srow, scol) of ints specifying the row and

column where the token begins in the source; a 2-tuple (erow, ecol) of

ints specifying the row and column where the token ends in the source; and

the line on which the token was found. The line passed (the last tuple item)

is the logical line; continuation lines are included. The 5 tuple is

returned as a named tuple with the field names:

type string start end line.

The returned named tuple has an additional property named

exact_type that contains the exact operator type for

OP tokens. For all other token ty