The Python Standard Library » 31. Python Language Services

The Python Standard Library »

31. Python Language Services

>>> for pp in dir(parser):
...     print pp
...
ASTType
ParserError
STType
__copyright__
__doc__
__file__
__name__
__package__
__version__
_pickler
ast2list
ast2tuple
compileast
compilest
expr


isexpr()
issuite()


sequence2ast
sequence2st
st2list
st2tuple
suite
tuple2ast
tuple2st







31.2. Abstract Syntax Trees
31.2.1. Node classes
class ast.AST

    This is the base of all AST node classes. The actual node classes are derived from the Parser/Python.asdl file, which is reproduced below. They are defined in the _ast C module and re-exported in ast.

    There is one class defined for each left-hand side symbol in the abstract grammar (for example, ast.stmt or ast.expr). In addition, there is one class defined for each constructor on the right-hand side; these classes inherit from the classes for the left-hand side trees. For example, ast.BinOp inherits from ast.expr. For production rules with alternatives (aka “sums”), the left-hand side class is abstract: only instances of specific constructor nodes are ever created.

    _fields

        Each concrete class has an attribute _fields which gives the names of all child nodes.

        Each instance of a concrete class has one attribute for each child node, of the type as defined in the grammar. For example, ast.BinOp instances have an attribute left of type ast.expr.

        If these attributes are marked as optional in the grammar (using a question mark), the value might be None. If the attributes can have zero-or-more values (marked with an asterisk), the values are represented as Python lists. All possible attributes must be present and have valid values when compiling an AST with compile().

    lineno
    col_offset
        Instances of ast.expr and ast.stmt subclasses have lineno and col_offset attributes. The lineno is the line number of source text (1-indexed so the first line is line 1) and the col_offset is the UTF-8 byte offset of the first token that generated the node. The UTF-8 offset is recorded because the parser uses UTF-8 internally.

    The constructor of a class ast.T parses its arguments as follows:

        If there are positional arguments, there must be as many as there are items in T._fields; they will be assigned as attributes of these names.
        If there are keyword arguments, they will set the attributes of the same names to the given values.

    For example, to create and populate an ast.UnaryOp node, you could use

    node = ast.UnaryOp()
    node.op = ast.USub()
    node.operand = ast.Num()
    node.operand.n = 5
    node.operand.lineno = 0
    node.operand.col_offset = 0
    node.lineno = 0
    node.col_offset = 0

    or the more compact

    node = ast.UnaryOp(ast.USub(), ast.Num(5, lineno=0, col_offset=0),
                       lineno=0, col_offset=0)

    New in version 2.6: The constructor as explained above was added. In Python 2.5 nodes had to be created by calling the class constructor without arguments and setting the attributes afterwards.








31.2.2. Abstract Grammar

The module defines a string constant __version__ which is the decimal Subversion revision number of the file shown below.

The abstract grammar is currently defined as follows:

-- ASDL's five builtin types are identifier, int, string, object, bool

module Python version "$Revision: 62047 $"
{
    mod = Module(stmt* body)
        | Interactive(stmt* body)
        | Expression(expr body)

        -- not really an actual node but useful in Jython's typesystem.
        | Suite(stmt* body)

    stmt = FunctionDef(identifier name, arguments args,
                            stmt* body, expr* decorator_list)
          | ClassDef(identifier name, expr* bases, stmt* body, expr *decorator_list)
          | Return(expr? value)

          | Delete(expr* targets)
          | Assign(expr* targets, expr value)
          | AugAssign(expr target, operator op, expr value)

          -- not sure if bool is allowed, can always use int
           | Print(expr? dest, expr* values, bool nl)

          -- use 'orelse' because else is a keyword in target languages
          | For(expr target, expr iter, stmt* body, stmt* orelse)
          | While(expr test, stmt* body, stmt* orelse)
          | If(expr test, stmt* body, stmt* orelse)
          | With(expr context_expr, expr? optional_vars, stmt* body)

          -- 'type' is a bad name
          | Raise(expr? type, expr? inst, expr? tback)
          | TryExcept(stmt* body, excepthandler* handlers, stmt* orelse)
          | TryFinally(stmt* body, stmt* finalbody)
          | Assert(expr test, expr? msg)

          | Import(alias* names)
          | ImportFrom(identifier module, alias* names, int? level)

          -- Doesn't capture requirement that locals must be
          -- defined if globals is
          -- still supports use as a function!
          | Exec(expr body, expr? globals, expr? locals)

          | Global(identifier* names)
          | Expr(expr value)
          | Pass | Break | Continue

          -- XXX Jython will be different
          -- col_offset is the byte offset in the utf8 string the parser uses
          attributes (int lineno, int col_offset)

          -- BoolOp() can use left & right?
    expr = BoolOp(boolop op, expr* values)
         | BinOp(expr left, operator op, expr right)
         | UnaryOp(unaryop op, expr operand)
         | Lambda(arguments args, expr body)
         | IfExp(expr test, expr body, expr orelse)
         | Dict(expr* keys, expr* values)
         | ListComp(expr elt, comprehension* generators)
         | GeneratorExp(expr elt, comprehension* generators)
         -- the grammar constrains where yield expressions can occur
         | Yield(expr? value)
         -- need sequences for compare to distinguish between
         -- x < 4 < 3 and (x < 4) < 3
         | Compare(expr left, cmpop* ops, expr* comparators)
         | Call(expr func, expr* args, keyword* keywords,
             expr? starargs, expr? kwargs)
         | Repr(expr value)
         | Num(object n) -- a number as a PyObject.
         | Str(string s) -- need to specify raw, unicode, etc?
         -- other literals? bools?

         -- the following expression can appear in assignment context
         | Attribute(expr value, identifier attr, expr_context ctx)
         | Subscript(expr value, slice slice, expr_context ctx)
         | Name(identifier id, expr_context ctx)
         | List(expr* elts, expr_context ctx)
         | Tuple(expr* elts, expr_context ctx)

          -- col_offset is the byte offset in the utf8 string the parser uses
          attributes (int lineno, int col_offset)

    expr_context = Load | Store | Del | AugLoad | AugStore | Param

    slice = Ellipsis | Slice(expr? lower, expr? upper, expr? step)
          | ExtSlice(slice* dims)
          | Index(expr value)

    boolop = And | Or

    operator = Add | Sub | Mult | Div | Mod | Pow | LShift
                 | RShift | BitOr | BitXor | BitAnd | FloorDiv

    unaryop = Invert | Not | UAdd | USub

    cmpop = Eq | NotEq | Lt | LtE | Gt | GtE | Is | IsNot | In | NotIn

    comprehension = (expr target, expr iter, expr* ifs)

    -- not sure what to call the first argument for raise and except
    excepthandler = ExceptHandler(expr? type, expr? name, stmt* body)
                    attributes (int lineno, int col_offset)

    arguments = (expr* args, identifier? vararg,
             identifier? kwarg, expr* defaults)

        -- keyword arguments supplied to call
        keyword = (identifier arg, expr value)

        -- import name with optional 'as' alias.
        alias = (identifier name, identifier? asname)
}












31.2.3. ast Helpers

New in version 2.6.

Apart from the node classes, ast module defines these utility functions and classes for traversing abstract syntax trees:

ast.parse(expr, filename='<unknown>', mode='exec')
ast.literal_eval(node_or_string)
ast.get_docstring(node, clean=True)
ast.fix_missing_locations(node)
ast.increment_lineno(node, n=1)
ast.copy_location(new_node, old_node)
ast.iter_fields(node)
ast.iter_child_nodes(node)
ast.walk(node)
class ast.NodeVisitor
class ast.NodeTransformer
ast.dump(node, annotate_fields=True, include_attributes=False)












31.3. symtable — Access to the compiler’s symbol tables
Symbol tables are generated by the compiler from AST just before bytecode is generated. The symbol table is responsible for calculating the scope of every identifier in the code. symtable provides an interface to examine these tables.


31.3.1. Generating Symbol Tables
symtable.symtable(code, filename, compile_type)



31.3.2. Examining Symbol Tables

class symtable.SymbolTable
    A namespace table for a block. The constructor is not public.

    get_type()
    get_id()
    get_name()
    get_lineno()
    is_optimized()
    is_nested()
    has_children()
    has_exec()
    has_import_star()
    get_identifiers()
    lookup(name)
    get_symbols()
    get_children()
    
    
    
class symtable.Function
    A namespace for a function or method. This class inherits SymbolTable.

    get_parameters()
    get_locals()
    get_globals()
    get_frees()
    
    
    
class symtable.Class
    A namespace of a class. This class inherits SymbolTable.

    get_methods()

class symtable.Symbol
    An entry in a SymbolTable corresponding to an identifier in the source. The constructor is not public.

    get_name()
    is_referenced()
    is_imported()
    is_parameter()
    is_global()
    is_declared_global()
    is_local()
    is_free()
    is_assigned()
    is_namespace()
    get_namespaces()
    get_namespace()




>>> for st in dir(symtable):
...     print st
...
CELL
Class
DEF_BOUND
DEF_GLOBAL
DEF_IMPORT
DEF_LOCAL
DEF_PARAM
FREE
Function
GLOBAL_EXPLICIT
GLOBAL_IMPLICIT
LOCAL
OPT_BARE_EXEC
OPT_EXEC
OPT_IMPORT_STAR
SCOPE_MASK
SCOPE_OFF
Symbol
SymbolTable
SymbolTableFactory
USE
__all__
__builtins__
__doc__
__file__
__name__
__package__
_newSymbolTable
_symtable
symtable
weakref
















31.4. symbol — Constants used with Python parse trees
    This module provides constants which represent the numeric values of
    internal nodes of the parse tree. Unlike most Python constants,
    these use lower-case names. Refer to the file Grammar/Grammar in the
    Python distribution for the definitions of the names in the context
    of the language grammar. The specific numeric values which the names
    map to may change between Python versions.


This module also provides one additional data object:
symbol.sym_name
    Dictionary mapping the numeric values of the constants defined in this
    module back to name strings, allowing more human-readable representati
    on of parse trees to be generated.


>>> for sy in dir(symbol):
...     print sy
...
__builtins__
__doc__
__file__
__name__
__package__
_name
_value
and_expr
and_test
arglist
argument
arith_expr
assert_stmt
atom
augassign
break_stmt
classdef
comp_for
comp_if
comp_iter
comp_op
comparison
compound_stmt
continue_stmt
decorated
decorator
decorators
del_stmt
dictorsetmaker
dotted_as_name
dotted_as_names
dotted_name
encoding_decl
eval_input
except_clause
exec_stmt
expr
expr_stmt
exprlist
factor
file_input
flow_stmt
for_stmt
fpdef
fplist
funcdef
global_stmt
if_stmt
import_as_name
import_as_names
import_from
import_name
import_stmt
lambdef
list_for
list_if
list_iter
listmaker
main
not_test
old_lambdef
old_test
or_test
parameters
pass_stmt
power
print_stmt
raise_stmt
return_stmt
shift_expr
simple_stmt
single_input
sliceop
small_stmt
stmt
subscript
subscriptlist
suite
sym_name
term
test
testlist
testlist1
testlist_comp
testlist_safe
trailer
try_stmt
varargslist
while_stmt
with_item
with_stmt
xor_expr
yield_expr
yield_stmt








31.5. token — Constants used with Python parse trees
    This module provides constants which represent the numeric values
    of leaf nodes of the parse tree (terminal tokens). Refer to the file
    Grammar/Grammar in the Python distribution for the definitions of
    the names in the context of the language grammar. The specific
    numeric values which the names map to may change between Python
    versions.

    This module also provides one data object and some functions. The
    functions mirror definitions in the Python C header files.

token.tok_name
    Dictionary mapping the numeric values of the constants defined in
    this module back to name strings, allowing more human-readable
    representation of parse trees to be generated.

token.ISTERMINAL(x)
token.ISNONTERMINAL(x)
token.ISEOF(x)



>>> for to in dir(token):
...     print to
...
AMPER
AMPEREQUAL
AT
BACKQUOTE
CIRCUMFLEX
CIRCUMFLEXEQUAL
COLON
COMMA
DEDENT
DOT
DOUBLESLASH
DOUBLESLASHEQUAL
DOUBLESTAR
DOUBLESTAREQUAL
ENDMARKER
EQEQUAL
EQUAL
ERRORTOKEN
GREATER
GREATEREQUAL
INDENT
ISEOF
ISNONTERMINAL
ISTERMINAL
LBRACE
LEFTSHIFT
LEFTSHIFTEQUAL
LESS
LESSEQUAL
LPAR
LSQB
MINEQUAL
MINUS
NAME
NEWLINE
NOTEQUAL
NT_OFFSET
NUMBER
N_TOKENS
OP
PERCENT
PERCENTEQUAL
PLUS
PLUSEQUAL
RBRACE
RIGHTSHIFT
RIGHTSHIFTEQUAL
RPAR
RSQB
SEMI
SLASH
SLASHEQUAL
STAR
STAREQUAL
STRING
TILDE
VBAR
VBAREQUAL
__builtins__
__doc__
__file__
__name__
__package__
main
tok_name










31.6. keyword — Testing for Python keywords
    This module allows a Python program to determine if a string is a
    keyword.

keyword.iskeyword(s)
keyword.kwlist
    Sequence containing all the keywords defined for the interpreter.
    If any keywords are defined to only be active when particular
    __future__ statements are in effect, these will be included as well.


>>> dir(keyword)
['__all__', '__builtins__', '__doc__', '__file__', '__name__',
'__package__', 'iskeyword', 'kwlist', 'main']













31.7. tokenize — Tokenizer for Python source
    The tokenize module provides a lexical scanner for Python source
    code, implemented in Python. The scanner in this module returns
    comments as tokens as well, making it useful for implementing
    “pretty-printers,” including colorizers for on-screen displays.
    
    The primary entry point is a generator:
tokenize.generate_tokens(readline)

    An older entry point is retained for backward compatibility:
tokenize.tokenize(readline[, tokeneater])
                    tokenize.COMMENT
                    tokenize.NL

    Another function is provided to reverse the tokenization process:
tokenize.untokenize(iterable)

>>> for tn in dir(tokenize):
...     print tn
...
AMPER
AMPEREQUAL
AT
BACKQUOTE
Binnumber
Bracket
CIRCUMFLEX
CIRCUMFLEXEQUAL
COLON
COMMA
COMMENT
Comment
ContStr
DEDENT
DOT
DOUBLESLASH
DOUBLESLASHEQUAL
DOUBLESTAR
DOUBLESTAREQUAL
Decnumber
Double
Double3
ENDMARKER
EQEQUAL
EQUAL
ERRORTOKEN
Expfloat
Exponent
Floatnumber
Funny
GREATER
GREATEREQUAL
Hexnumber
INDENT
ISEOF
ISNONTERMINAL
ISTERMINAL
Ignore
Imagnumber
Intnumber
LBRACE
LEFTSHIFT
LEFTSHIFTEQUAL
LESS
LESSEQUAL
LPAR
LSQB
MINEQUAL
MINUS
NAME
NEWLINE
NL
NOTEQUAL
NT_OFFSET
NUMBER
N_TOKENS
Name
Number
OP
Octnumber
Operator
PERCENT
PERCENTEQUAL
PLUS
PLUSEQUAL
PlainToken
Pointfloat
PseudoExtras
PseudoToken
RBRACE
RIGHTSHIFT
RIGHTSHIFTEQUAL
RPAR
RSQB
SEMI
SLASH
SLASHEQUAL
STAR
STAREQUAL
STRING
Single
Single3
Special
StopTokenizing
String
TILDE
Token
TokenError
Triple
Untokenizer
VBAR
VBAREQUAL
Whitespace
__all__
__author__
__builtins__
__credits__
__doc__
__file__
__name__
__package__
any
double3prog
endprogs
generate_tokens
group
main
maybe
printtoken
pseudoprog
re
single3prog
single_quoted
string
t
tabsize
tok_name
tokenize
tokenize_loop
tokenprog
triple_quoted
untokenize






31.9. pyclbr — Python class browser support
pyclbr.readmodule(module[, path=None])
pyclbr.readmodule_ex(module[, path=None])

31.9.1. Class Objects
Class.module
Class.name
Class.super
Class.methods
Class.file
Class.lineno

31.9.2. Function Objects
Function.module
Function.name
Function.file
Function.lineno



>>> for pc in dir(pyclbr):
...     print pc
...
Class
DEDENT
Function
NAME
OP
__all__
__builtins__
__doc__
__file__
__name__
__package__
_getname
_getnamelist
_main
_modules
_readmodule
imp
itemgetter
readmodule
readmodule_ex
sys
tokenize

-------------------------------end of file------------------