modulegraph/doc/modulegraph.rst

:mod:`modulegraph.modulegraph` --- Find modules used by a script
================================================================

.. module:: modulegraph.modulegraph
   :synopsis: Find modules used by a script

This module defines :class:`ModuleGraph`, which is used to find
the dependencies of scripts using bytecode analysis.

A number of APIs in this module refer to filesystem path. Those paths can refer to
files inside zipfiles (for example when there are zipped egg files on :data:`sys.path`).
Filenames referring to entries in a zipfile are not marked any way, if ``"somepath.zip"``
refers to a zipfile, that is ``"somepath.zip/embedded/file"`` will be used to refer to
``embedded/file`` inside the zipfile.

The actual graph
----------------

.. class:: ModuleGraph([path[, excludes[, replace_paths[, implies[, graph[, debug]]]]]])

   Create a new ModuleGraph object. Use the :meth:`run_script` method to add scripts,
   and their dependencies to the graph.

   :param path: Python search path to use, defaults to :data:`sys.path`
   :param excludes: Iterable with module names that should not be included as a dependency
   :param replace_paths: List of pathname rewrites ``(old, new)``. When this argument is
     supplied the ``co_filename`` attributes of code objects get rewritten before scanning
     them for dependencies.
   :param implies: Implied module dependencies, a mapping from a module name to the list
     of modules it depends on. Use this to tell modulegraph about dependencies that cannot
     be found by code inspection (such as imports from C code or using the :func:`__import__`
     function).
   :param graph: A precreated :class:`Graph <altgraph.Graph.Graph>` object to use, the
     default is to create a new one.
   :param debug: The :class:`ObjectGraph <altgraph.ObjectGraph.ObjectGraph>` debug level.


.. method:: run_script(pathname[, caller])

   Create, and return,  a node by path (not module name). The *pathname* should
   refer to a Python source file and will be scanned for dependencies.

   The optional argument *caller* is the the node that calls this script,
   and is used to add a reference in the graph.

.. method:: import_hook(name[[, caller[, fromlist[, level, [, attr]]]])

   Import a module and analyse its dependencies

   :arg name:     The module name
   :arg caller:   The node that caused the import to happen
   :arg fromlist: The list of names to import, this is an empty list for
      ``import name`` and a list of names for ``from name import a, b, c``.
   :arg level:    The import level. The value should be ``-1`` for classical Python 2
     imports, ``0`` for absolute imports and a positive number for relative imports (
     where the value is the number of leading dots in the imported name).
   :arg attr:     Attributes for the graph edge.


.. method:: implyNodeReference(node, other, edgeData=None)

   Explictly mark that *node* depends on *other*. Other is either
   a :class:`node <Node>` or the name of a module that will be
   searched for as if it were an absolute import.


.. method:: createReference(fromnode, tonode[, edge_data])

   Create a reference from *fromnode* to *tonode*, with optional edge data.

   The default for *edge_data* is ``"direct"``.

.. method:: getReferences(fromnode)

   Yield all nodes that *fromnode* refers to. That is, all modules imported
   by *fromnode*.

   Node :data:`None` is the root of the graph, and refers to all notes that were
   explicitly imported by :meth:`run_script` or :meth:`import_hook`, unless you use
   an explicit parent with those methods.

   .. versionadded:: 0.11

.. method:: getReferers(tonode, collapse_missing_modules=True)

   Yield all nodes that refer to *tonode*. That is, all modules that import
   *tonode*.

   If *collapse_missing_modules* is false this includes refererences from
   :class:`MissingModule` nodes, otherwise :class:`MissingModule` nodes
   are replaced by the "real" nodes that reference this missing node.

   .. versionadded:: 0.12

.. method:: foldReferences(pkgnode)

   Hide all submodule nodes for package *pkgnode* and add ingoing and outgoing
   edges to *pkgnode* based on the edges from the submodule nodes.

   This can be used to simplify a module graph: after folding 'email' all
   references to modules in the 'email' package are references to the package.

   .. versionadded: 0.11

.. method:: findNode(name)

   Find a node by identifier.  If a node by that identifier exists, it will be returned.

   If a lazy node exists by that identifier with no dependencies (excluded), it will be
   instantiated and returned.

   If a lazy node exists by that identifier with dependencies, it and its
   dependencies will be instantiated and scanned for additional depende


.. method:: create_xref([out])

   Write an HTML file to the *out* stream (defaulting to :data:`sys.stdout`).

   The HTML file contains a textual description of the dependency graph.


.. method:: graphreport([fileobj[, flatpackages]])

   .. todo:: To be documented


.. method:: report()

   Print a report to stdout, listing the found modules with their
   paths, as well as modules that are missing, or seem to be missing.


Mostly internal methods
.......................

The methods in this section should be considered as methods for subclassing at best,
please let us know if you need these methods in your code as they are on track to be
made private methods before the 1.0 release.

.. warning:: The methods in this section will be refactored in a future release,
   the current architecture makes it unnecessarily hard to write proper tests.

.. method:: determine_parent(caller)

   Returns the node of the package root voor *caller*. If *caller* is a package
   this is the node itself, if the node is a module in a package this is the
   node of for the package and otherwise the *caller* is not a package and
   the result is :data:`None`.

.. method:: find_head_package(parent, name[, level])

   .. todo:: To be documented


.. method:: load_tail(mod, tail)

   This method is called to load the rest of a dotted name after loading the root
   of a package. This will import all intermediate modules as well (using
   :meth:`import_module`), and returns the module :class:`node <Node>` for the
   requested node.

   .. note:: When *tail* is empty this will just return *mod*.

   :arg mod:   A start module (instance of :class:`Node`)
   :arg tail:  The rest of a dotted name, can be empty
   :raise ImportError: When the requested (or one of its parents) module cannot be found
   :returns: the requested module


.. method:: ensure_fromlist(m, fromlist)

   Yield all submodules that would be imported when importing *fromlist*
   from *m* (using ``from m import fromlist...``).

   *m* must be a package and not a regular module.

.. method:: find_all_submodules(m)

   Yield the filenames for submodules of in the same package as *m*.


.. method:: import_module(partname, fqname, parent)

   Perform import of the module with basename *partname* (``path``) and
   full name *fqname* (``os.path``). Import is performed by *parent*.

   This will create a reference from the parent node to the
   module node and will load the module node when it is not already
   loaded.


.. method:: load_module(fqname, fp, pathname, (suffix, mode, type))

   Load the module named *fqname* from the given *pathame*. The
   argument *fp* is either :data:`None`, or a stream where the
   code for the Python module can be loaded (either byte-code or
   the source code). The *(suffix, mode, type)* tuple are the
   suffix of the source file, the open mode for the file and the
   type of module.

   Creates a node of the right class and processes the dependencies
   of the :class:`node <Node>` by scanning the byte-code for the node.

   Returns the resulting :class:`node <Node>`.


.. method:: scan_code(code, m)

   Scan the *code* object for module *m* and update the dependencies of
   *m* using the import statemets found in the code.

   This will automaticly scan the code for nested functions, generator
   expressions and list comprehensions as well.


.. method:: load_package(fqname, pathname)

   Load a package directory.


.. method:: find_module(name, path[, parent])

   Locates a module named *name* that is not yet part of the
   graph. This method will raise :exc:`ImportError` when
   the module cannot be found or when it is already part
   of the graph. The *name* can not be a dotted name.

   The *path* is the search path used, or :data:`None` to
   use the default path.

   When the *parent* is specified *name* refers to a
   subpackage of *parent*, and *path* should be the
   search path of the parent.

   Returns the result of the global function
   :func:`find_module <modulegraph.modulegraph.find_module>`.


.. method:: itergraphreport([name[, flatpackages]])

   .. todo:: To be documented


.. method:: replace_paths_in_code(co)

   Replace the filenames in code object *co* using the *replace_paths* value that
   was passed to the contructor. Returns the rewritten code object.


.. method:: calc_setuptools_nspackages()

   Returns a mapping from package name to a list of paths where that package
   can be found in ``--single-version-externally-managed`` form.

   This method is used to be able to find those packages: these use
   a magic ``.pth`` file to ensure that the package is added to :data:`sys.path`,
   as they do not contain an ``___init__.py`` file.

   Packages in this form are used by system packages and the "pip"
   installer.


Graph nodes
-----------

The :class:`ModuleGraph` contains nodes that represent the various types of modules.

.. class:: Alias(value)

   This is a subclass of string that is used to mark module aliases.


.. class:: Node(identifier)

   Base class for nodes, which provides the common functionality.

   Nodes can by used as mappings for storing arbitrary data in the node.

   Nodes are compared by comparing their *identifier*.

.. data:: debug

   Debug level (integer)

.. data:: graphident

   The node identifier, this is the value of the *identifier* argument
   to the constructor.

.. data:: identifier

   The node identifier, this is the value of the *identifier* argument
   to the constructor.

.. data:: filename

   The filename associated with this node.

.. data:: packagepath

   The value of ``__path__`` for this node.

.. data:: code

   The :class:`code object <types.CodeObject>` associated with this node

.. data:: globalnames

   The set of global names that are assigned to in this module. This
   includes those names imported through startimports of Python modules.

.. data:: startimports

   The set of startimports this module did that could not be resolved,
   ie. a startimport from a non-Python module.


.. method:: __contains__(name)

   Return if there is a value associated with *name*.

   This method is usually accessed as ``name in aNode``.

.. method:: __setitem__(name, value)

   Set the value of *name* to *value*.

   This method is usually accessed as ``aNode[name] = value``.

.. method:: __getitem__(name)

   Returns the value of *name*, raises :exc:`KeyError` when
   it cannot be found.

   This method is usually accessed as ``value = aNode[name]``.

.. method:: get(name[, default])

   Returns the value of *name*, or the default value when it
   cannot be found. The *default* is :data:`None` when not specified.

.. method:: infoTuple()

   Returns a tuple with information used in the :func:`repr`
   output for the node. Subclasses can add additional informations
   to the result.


.. class:: AliasNode (name, node)

   A node that represents an alias from a name to another node.

   The value of attribute *graphident* for this node will be the
   value of *name*, the other :class:`Node` attributed are
   references to those attributed in *node*.

.. class:: BadModule(identifier)

   Base class for nodes that should be ignored for some reason

.. class:: ExcludedModule(identifier)

   A module that is explicitly excluded.

.. class:: MissingModule(identifier)

   A module that is imported but cannot be located.


.. class:: Script(filename)

   A python script.

   .. data:: filename

      The filename for the script

.. class:: BaseModule(name[, filename[, path]])

    The base class for actual modules. The *name* is
    the possibly dotted module name, *filename* is the
    filesystem path to the module and *path* is the
    value of ``__path__`` for the module.

.. data:: graphident

   The name of the module

.. data:: filename

   The filesystem path to the module.

.. data:: path

   The value of ``__path__`` for this module.

.. class:: BuiltinModule(name)

   A built-in module (on in :data:`sys.builtin_module_names`).

.. class:: SourceModule(name)

   A module for which the python source code is available.

.. class:: InvalidSourceModule(name)

   A module for which the python source code is available, but where
   that source code cannot be compiled (due to syntax errors).

   This is a subclass of :class:`SourceModule`.

   .. versionadded:: 0.12

.. class:: CompiledModule(name)

   A module for which only byte-code is available.

.. class:: Package(name)

   Represents a python package

.. class:: NamespacePackage(name)

   Represents a python namespace package.

   This is a subclass of :class:`Package`.

.. class:: Extension(name)

   A native extension


.. warning:: A number of other node types are defined in the module. Those modules aren't
   used by modulegraph and will be removed in a future version.


Edge data
---------

The edges in a module graph by default contain information about the edge, represented
by an instance of :class:`DependencyInfo`.

.. class:: DependencyInfo(conditional, function, tryexcept, fromlist)

   This class is a :func:`namedtuple <collections.namedtuple>` for representing
   the information on a dependency between two modules.

   All attributes can be used to deduce if a dependency is essential or not, and
   are particularly useful when reporting on missing modules (dependencies on
   :class:`MissingModule`).

   .. data:: fromlist

      A boolean that is true iff the target of the edge is named in the "import"
      list of a "from" import ("from package import module").

      When the target module is imported multiple times this attribute is false
      unless all imports are in "import" list of a "from" import.

   .. data:: function

      A boolean that is true iff the import is done inside a function definition,
      and is false for imports in module scope (or class scope for classes that
      aren't definined in a function).

   .. data:: tryexcept

      A boolean that is true iff the import that is done in the "try" or "except"
      block of a try statement (but not in the "else" block).

   .. data:: conditional

      A boolean that is true iff the import is done in either block of an "if"
      statement.

   When the target of the edge is imported multiple times the :data:`function`,
   :data:`tryexcept` and :data:`conditional` attributes of all imports are
   merged: when there is an import where all these attributes are false the
   attributes are false, otherwise each attribute is set to true if it is
   true for at least one of the imports.

   For example, when a module is imported both in a try-except statement and
   furthermore is imported in a function (in two separate statements),
   both :data:`tryexcept` and :data:`function` will be true.  But if there
   is a third unconditional toplevel import for that module as well all
   three attributes are false.

   .. warning::

      All attributes but :data:`fromlist` will be false when the source of
      a dependency is scanned from a byte-compiled module instead of a python
      source file. The :data:`fromlist` attribute will stil be set correctly.

Utility functions
-----------------

.. function:: find_module(name[, path])

   A version of :func:`imp.find_module` that works with zipped packages (and other
   :pep:`302` importers).

.. function:: moduleInfoForPath(path)

   Return the module name, readmode and type for the file at *path*, or
   None if it doesn't seem to be a valid module (based on its name).

.. function:: addPackagePath(packagename, path)

   Add *path* to the value of ``__path__`` for the package named *packagename*.

.. function:: replacePackage(oldname, newname)

   Rename *oldname* to *newname* when it is found by the module finder. This
   is used as a workaround for the hack that the ``_xmlplus`` package uses
   to inject itself in the ``xml`` namespace.