1:mod:`ast` --- Abstract Syntax Trees 2==================================== 3 4.. module:: ast 5 :synopsis: Abstract Syntax Tree classes and manipulation. 6 7.. sectionauthor:: Martin v. Löwis <martin@v.loewis.de> 8.. sectionauthor:: Georg Brandl <georg@python.org> 9 10**Source code:** :source:`Lib/ast.py` 11 12-------------- 13 14The :mod:`ast` module helps Python applications to process trees of the Python 15abstract syntax grammar. The abstract syntax itself might change with each 16Python release; this module helps to find out programmatically what the current 17grammar looks like. 18 19An abstract syntax tree can be generated by passing :data:`ast.PyCF_ONLY_AST` as 20a flag to the :func:`compile` built-in function, or using the :func:`parse` 21helper provided in this module. The result will be a tree of objects whose 22classes all inherit from :class:`ast.AST`. An abstract syntax tree can be 23compiled into a Python code object using the built-in :func:`compile` function. 24 25 26Node classes 27------------ 28 29.. class:: AST 30 31 This is the base of all AST node classes. The actual node classes are 32 derived from the :file:`Parser/Python.asdl` file, which is reproduced 33 :ref:`below <abstract-grammar>`. They are defined in the :mod:`_ast` C 34 module and re-exported in :mod:`ast`. 35 36 There is one class defined for each left-hand side symbol in the abstract 37 grammar (for example, :class:`ast.stmt` or :class:`ast.expr`). In addition, 38 there is one class defined for each constructor on the right-hand side; these 39 classes inherit from the classes for the left-hand side trees. For example, 40 :class:`ast.BinOp` inherits from :class:`ast.expr`. For production rules 41 with alternatives (aka "sums"), the left-hand side class is abstract: only 42 instances of specific constructor nodes are ever created. 43 44 .. index:: single: ? (question mark); in AST grammar 45 .. index:: single: * (asterisk); in AST grammar 46 47 .. attribute:: _fields 48 49 Each concrete class has an attribute :attr:`_fields` which gives the names 50 of all child nodes. 51 52 Each instance of a concrete class has one attribute for each child node, 53 of the type as defined in the grammar. For example, :class:`ast.BinOp` 54 instances have an attribute :attr:`left` of type :class:`ast.expr`. 55 56 If these attributes are marked as optional in the grammar (using a 57 question mark), the value might be ``None``. If the attributes can have 58 zero-or-more values (marked with an asterisk), the values are represented 59 as Python lists. All possible attributes must be present and have valid 60 values when compiling an AST with :func:`compile`. 61 62 .. attribute:: lineno 63 col_offset 64 end_lineno 65 end_col_offset 66 67 Instances of :class:`ast.expr` and :class:`ast.stmt` subclasses have 68 :attr:`lineno`, :attr:`col_offset`, :attr:`lineno`, and :attr:`col_offset` 69 attributes. The :attr:`lineno` and :attr:`end_lineno` are the first and 70 last line numbers of source text span (1-indexed so the first line is line 1) 71 and the :attr:`col_offset` and :attr:`end_col_offset` are the corresponding 72 UTF-8 byte offsets of the first and last tokens that generated the node. 73 The UTF-8 offset is recorded because the parser uses UTF-8 internally. 74 75 Note that the end positions are not required by the compiler and are 76 therefore optional. The end offset is *after* the last symbol, for example 77 one can get the source segment of a one-line expression node using 78 ``source_line[node.col_offset : node.end_col_offset]``. 79 80 The constructor of a class :class:`ast.T` parses its arguments as follows: 81 82 * If there are positional arguments, there must be as many as there are items 83 in :attr:`T._fields`; they will be assigned as attributes of these names. 84 * If there are keyword arguments, they will set the attributes of the same 85 names to the given values. 86 87 For example, to create and populate an :class:`ast.UnaryOp` node, you could 88 use :: 89 90 node = ast.UnaryOp() 91 node.op = ast.USub() 92 node.operand = ast.Constant() 93 node.operand.value = 5 94 node.operand.lineno = 0 95 node.operand.col_offset = 0 96 node.lineno = 0 97 node.col_offset = 0 98 99 or the more compact :: 100 101 node = ast.UnaryOp(ast.USub(), ast.Constant(5, lineno=0, col_offset=0), 102 lineno=0, col_offset=0) 103 104.. deprecated:: 3.8 105 106 Class :class:`ast.Constant` is now used for all constants. Old classes 107 :class:`ast.Num`, :class:`ast.Str`, :class:`ast.Bytes`, 108 :class:`ast.NameConstant` and :class:`ast.Ellipsis` are still available, 109 but they will be removed in future Python releases. 110 111 112.. _abstract-grammar: 113 114Abstract Grammar 115---------------- 116 117The abstract grammar is currently defined as follows: 118 119.. literalinclude:: ../../Parser/Python.asdl 120 :language: none 121 122 123:mod:`ast` Helpers 124------------------ 125 126Apart from the node classes, the :mod:`ast` module defines these utility functions 127and classes for traversing abstract syntax trees: 128 129.. function:: parse(source, filename='<unknown>', mode='exec', *, type_comments=False, feature_version=None) 130 131 Parse the source into an AST node. Equivalent to ``compile(source, 132 filename, mode, ast.PyCF_ONLY_AST)``. 133 134 If ``type_comments=True`` is given, the parser is modified to check 135 and return type comments as specified by :pep:`484` and :pep:`526`. 136 This is equivalent to adding :data:`ast.PyCF_TYPE_COMMENTS` to the 137 flags passed to :func:`compile()`. This will report syntax errors 138 for misplaced type comments. Without this flag, type comments will 139 be ignored, and the ``type_comment`` field on selected AST nodes 140 will always be ``None``. In addition, the locations of ``# type: 141 ignore`` comments will be returned as the ``type_ignores`` 142 attribute of :class:`Module` (otherwise it is always an empty list). 143 144 In addition, if ``mode`` is ``'func_type'``, the input syntax is 145 modified to correspond to :pep:`484` "signature type comments", 146 e.g. ``(str, int) -> List[str]``. 147 148 Also, setting ``feature_version`` to a tuple ``(major, minor)`` 149 will attempt to parse using that Python version's grammar. 150 Currently ``major`` must equal to ``3``. For example, setting 151 ``feature_version=(3, 4)`` will allow the use of ``async`` and 152 ``await`` as variable names. The lowest supported version is 153 ``(3, 4)``; the highest is ``sys.version_info[0:2]``. 154 155 .. warning:: 156 It is possible to crash the Python interpreter with a 157 sufficiently large/complex string due to stack depth limitations 158 in Python's AST compiler. 159 160 .. versionchanged:: 3.8 161 Added ``type_comments``, ``mode='func_type'`` and ``feature_version``. 162 163 164.. function:: literal_eval(node_or_string) 165 166 Safely evaluate an expression node or a string containing a Python literal or 167 container display. The string or node provided may only consist of the 168 following Python literal structures: strings, bytes, numbers, tuples, lists, 169 dicts, sets, booleans, and ``None``. 170 171 This can be used for safely evaluating strings containing Python values from 172 untrusted sources without the need to parse the values oneself. It is not 173 capable of evaluating arbitrarily complex expressions, for example involving 174 operators or indexing. 175 176 .. warning:: 177 It is possible to crash the Python interpreter with a 178 sufficiently large/complex string due to stack depth limitations 179 in Python's AST compiler. 180 181 .. versionchanged:: 3.2 182 Now allows bytes and set literals. 183 184 185.. function:: get_docstring(node, clean=True) 186 187 Return the docstring of the given *node* (which must be a 188 :class:`FunctionDef`, :class:`AsyncFunctionDef`, :class:`ClassDef`, 189 or :class:`Module` node), or ``None`` if it has no docstring. 190 If *clean* is true, clean up the docstring's indentation with 191 :func:`inspect.cleandoc`. 192 193 .. versionchanged:: 3.5 194 :class:`AsyncFunctionDef` is now supported. 195 196 197.. function:: get_source_segment(source, node, *, padded=False) 198 199 Get source code segment of the *source* that generated *node*. 200 If some location information (:attr:`lineno`, :attr:`end_lineno`, 201 :attr:`col_offset`, or :attr:`end_col_offset`) is missing, return ``None``. 202 203 If *padded* is ``True``, the first line of a multi-line statement will 204 be padded with spaces to match its original position. 205 206 .. versionadded:: 3.8 207 208 209.. function:: fix_missing_locations(node) 210 211 When you compile a node tree with :func:`compile`, the compiler expects 212 :attr:`lineno` and :attr:`col_offset` attributes for every node that supports 213 them. This is rather tedious to fill in for generated nodes, so this helper 214 adds these attributes recursively where not already set, by setting them to 215 the values of the parent node. It works recursively starting at *node*. 216 217 218.. function:: increment_lineno(node, n=1) 219 220 Increment the line number and end line number of each node in the tree 221 starting at *node* by *n*. This is useful to "move code" to a different 222 location in a file. 223 224 225.. function:: copy_location(new_node, old_node) 226 227 Copy source location (:attr:`lineno`, :attr:`col_offset`, :attr:`end_lineno`, 228 and :attr:`end_col_offset`) from *old_node* to *new_node* if possible, 229 and return *new_node*. 230 231 232.. function:: iter_fields(node) 233 234 Yield a tuple of ``(fieldname, value)`` for each field in ``node._fields`` 235 that is present on *node*. 236 237 238.. function:: iter_child_nodes(node) 239 240 Yield all direct child nodes of *node*, that is, all fields that are nodes 241 and all items of fields that are lists of nodes. 242 243 244.. function:: walk(node) 245 246 Recursively yield all descendant nodes in the tree starting at *node* 247 (including *node* itself), in no specified order. This is useful if you only 248 want to modify nodes in place and don't care about the context. 249 250 251.. class:: NodeVisitor() 252 253 A node visitor base class that walks the abstract syntax tree and calls a 254 visitor function for every node found. This function may return a value 255 which is forwarded by the :meth:`visit` method. 256 257 This class is meant to be subclassed, with the subclass adding visitor 258 methods. 259 260 .. method:: visit(node) 261 262 Visit a node. The default implementation calls the method called 263 :samp:`self.visit_{classname}` where *classname* is the name of the node 264 class, or :meth:`generic_visit` if that method doesn't exist. 265 266 .. method:: generic_visit(node) 267 268 This visitor calls :meth:`visit` on all children of the node. 269 270 Note that child nodes of nodes that have a custom visitor method won't be 271 visited unless the visitor calls :meth:`generic_visit` or visits them 272 itself. 273 274 Don't use the :class:`NodeVisitor` if you want to apply changes to nodes 275 during traversal. For this a special visitor exists 276 (:class:`NodeTransformer`) that allows modifications. 277 278 .. deprecated:: 3.8 279 280 Methods :meth:`visit_Num`, :meth:`visit_Str`, :meth:`visit_Bytes`, 281 :meth:`visit_NameConstant` and :meth:`visit_Ellipsis` are deprecated 282 now and will not be called in future Python versions. Add the 283 :meth:`visit_Constant` method to handle all constant nodes. 284 285 286.. class:: NodeTransformer() 287 288 A :class:`NodeVisitor` subclass that walks the abstract syntax tree and 289 allows modification of nodes. 290 291 The :class:`NodeTransformer` will walk the AST and use the return value of 292 the visitor methods to replace or remove the old node. If the return value 293 of the visitor method is ``None``, the node will be removed from its 294 location, otherwise it is replaced with the return value. The return value 295 may be the original node in which case no replacement takes place. 296 297 Here is an example transformer that rewrites all occurrences of name lookups 298 (``foo``) to ``data['foo']``:: 299 300 class RewriteName(NodeTransformer): 301 302 def visit_Name(self, node): 303 return copy_location(Subscript( 304 value=Name(id='data', ctx=Load()), 305 slice=Index(value=Constant(value=node.id)), 306 ctx=node.ctx 307 ), node) 308 309 Keep in mind that if the node you're operating on has child nodes you must 310 either transform the child nodes yourself or call the :meth:`generic_visit` 311 method for the node first. 312 313 For nodes that were part of a collection of statements (that applies to all 314 statement nodes), the visitor may also return a list of nodes rather than 315 just a single node. 316 317 Usually you use the transformer like this:: 318 319 node = YourTransformer().visit(node) 320 321 322.. function:: dump(node, annotate_fields=True, include_attributes=False) 323 324 Return a formatted dump of the tree in *node*. This is mainly useful for 325 debugging purposes. If *annotate_fields* is true (by default), 326 the returned string will show the names and the values for fields. 327 If *annotate_fields* is false, the result string will be more compact by 328 omitting unambiguous field names. Attributes such as line 329 numbers and column offsets are not dumped by default. If this is wanted, 330 *include_attributes* can be set to true. 331 332.. seealso:: 333 334 `Green Tree Snakes <https://greentreesnakes.readthedocs.io/>`_, an external documentation resource, has good 335 details on working with Python ASTs. 336