mpi/doc/python.qbk

[section:python Python Bindings]
[python]

Boost.MPI provides an alternative MPI interface from the _Python_
programming language via the `boost.mpi` module. The
Boost.MPI Python bindings, built on top of the C++ Boost.MPI using the
_BoostPython_ library, provide nearly all of the functionality of
Boost.MPI within a dynamic, object-oriented language.

The Boost.MPI Python module can be built and installed from the
`libs/mpi/build` directory. Just follow the [link
mpi.getting_started.config configuration] and [link mpi.getting_started.config.installation
installation] instructions for the C++ Boost.MPI. Once you have
installed the Python module, be sure that the installation location is
in your `PYTHONPATH`.

[section:quickstart Quickstart]

[python]

Getting started with the Boost.MPI Python module is as easy as
importing `boost.mpi`. Our first "Hello, World!" program is
just two lines long:

  import boost.mpi as mpi
  print "I am process %d of %d." % (mpi.rank, mpi.size)

Go ahead and run this program with several processes. Be sure to
invoke the `python` interpreter from `mpirun`, e.g.,

[pre
mpirun -np 5 python hello_world.py
]

This will return output such as:

[pre
I am process 1 of 5.
I am process 3 of 5.
I am process 2 of 5.
I am process 4 of 5.
I am process 0 of 5.
]

Point-to-point operations in Boost.MPI have nearly the same syntax in
Python as in C++. We can write a simple two-process Python program
that prints "Hello, world!" by transmitting Python strings:

  import boost.mpi as mpi

  if mpi.world.rank == 0:
    mpi.world.send(1, 0, 'Hello')
    msg = mpi.world.recv(1, 1)
    print msg,'!'
  else:
    msg = mpi.world.recv(0, 0)
    print (msg + ', '),
    mpi.world.send(0, 1, 'world')

There are only a few notable differences between this Python code and
the example [link mpi.tutorial.point_to_point in the C++
tutorial]. First of all, we don't need to write any initialization
code in Python: just loading the `boost.mpi` module makes the
appropriate `MPI_Init` and `MPI_Finalize` calls. Second, we're passing
Python objects from one process to another through MPI. Any Python
object that can be pickled can be transmitted; the next section will
describe in more detail how the Boost.MPI Python layer transmits
objects. Finally, when we receive objects with `recv`, we don't need
to specify the type because transmission of Python objects is
polymorphic.

When experimenting with Boost.MPI in Python, don't forget that help is
always available via `pydoc`: just pass the name of the module or
module entity on the command line (e.g., `pydoc
boost.mpi.communicator`) to receive complete reference
documentation. When in doubt, try it!
[endsect:quickstart]

[section:user_data Transmitting User-Defined Data]
Boost.MPI can transmit user-defined data in several different ways.
Most importantly, it can transmit arbitrary _Python_ objects by pickling
them at the sender and unpickling them at the receiver, allowing
arbitrarily complex Python data structures to interoperate with MPI.

Boost.MPI also supports efficient serialization and transmission of
C++ objects (that have been exposed to Python) through its C++
interface. Any C++ type that provides (de-)serialization routines that
meet the requirements of the Boost.Serialization library is eligible
for this optimization, but the type must be registered in advance. To
register a C++ type, invoke the C++ function [funcref
boost::mpi::python::register_serialized
register_serialized]. If your C++ types come from other Python modules
(they probably will!), those modules will need to link against the
`boost_mpi` and `boost_mpi_python` libraries as described in the [link
mpi.getting_started.config.installation installation section]. Note that you do
*not* need to link against the Boost.MPI Python extension module.

Finally, Boost.MPI supports separation of the structure of an object
from the data it stores, allowing the two pieces to be transmitted
separately. This "skeleton/content" mechanism, described in more
detail in a later section, is a communication optimization suitable
for problems with fixed data structures whose internal data changes
frequently.
[endsect:user_data]

[section:collectives Collectives]

Boost.MPI supports all of the MPI collectives (`scatter`, `reduce`,
`scan`, `broadcast`, etc.) for any type of data that can be
transmitted with the point-to-point communication operations. For the
MPI collectives that require a user-specified operation (e.g., `reduce`
and `scan`), the operation can be an arbitrary Python function. For
instance, one could concatenate strings with `all_reduce`:

  mpi.all_reduce(my_string, lambda x,y: x + y)

The following module-level functions implement MPI collectives:
  all_gather    Gather the values from all processes.
  all_reduce    Combine the results from all processes.
  all_to_all    Every process sends data to every other process.
  broadcast     Broadcast data from one process to all other processes.
  gather        Gather the values from all processes to the root.
  reduce        Combine the results from all processes to the root.
  scan          Prefix reduction of the values from all processes.
  scatter       Scatter the values stored at the root to all processes.
[endsect:collectives]

[section:skeleton_content Skeleton/Content Mechanism]
Boost.MPI provides a skeleton/content mechanism that allows the
transfer of large data structures to be split into two separate stages,
with the skeleton (or, "shape") of the data structure sent first and
the content (or, "data") of the data structure sent later, potentially
several times, so long as the structure has not changed since the
skeleton was transferred. The skeleton/content mechanism can improve
performance when the data structure is large and its shape is fixed,
because while the skeleton requires serialization (it has an unknown
size), the content transfer is fixed-size and can be done without
extra copies.

To use the skeleton/content mechanism from Python, you must first
register the type of your data structure with the skeleton/content
mechanism *from C++*. The registration function is [funcref
boost::mpi::python::register_skeleton_and_content
register_skeleton_and_content] and resides in the [headerref
boost/mpi/python.hpp <boost/mpi/python.hpp>] header.

Once you have registered your C++ data structures, you can extract
the skeleton for an instance of that data structure with `skeleton()`.
The resulting `skeleton_proxy` can be transmitted via the normal send
routine, e.g.,

  mpi.world.send(1, 0, skeleton(my_data_structure))

`skeleton_proxy` objects can be received on the other end via `recv()`,
which stores a newly-created instance of your data structure with the
same "shape" as the sender in its `"object"` attribute:

  shape = mpi.world.recv(0, 0)
  my_data_structure = shape.object

Once the skeleton has been transmitted, the content (accessed via
`get_content`) can be transmitted in much the same way. Note, however,
that the receiver also specifies `get_content(my_data_structure)` in its
call to receive:

  if mpi.rank == 0:
    mpi.world.send(1, 0, get_content(my_data_structure))
  else:
    mpi.world.recv(0, 0, get_content(my_data_structure))

Of course, this transmission of content can occur repeatedly, if the
values in the data structure--but not its shape--changes.

The skeleton/content mechanism is a structured way to exploit the
interaction between custom-built MPI datatypes and `MPI_BOTTOM`, to
eliminate extra buffer copies.
[endsect:skeleton_content]

[section:compatibility C++/Python MPI Compatibility]
Boost.MPI is a C++ library whose facilities have been exposed to Python
via the Boost.Python library. Since the Boost.MPI Python bindings are
build directly on top of the C++ library, and nearly every feature of
C++ library is available in Python, hybrid C++/Python programs using
Boost.MPI can interact, e.g., sending a value from Python but receiving
that value in C++ (or vice versa). However, doing so requires some
care. Because Python objects are dynamically typed, Boost.MPI transfers
type information along with the serialized form of the object, so that
the object can be received even when its type is not known. This
mechanism differs from its C++ counterpart, where the static types of
transmitted values are always known.

The only way to communicate between the C++ and Python views on
Boost.MPI is to traffic entirely in Python objects. For Python, this
is the normal state of affairs, so nothing will change. For C++, this
means sending and receiving values of type `boost::python::object`,
from the _BoostPython_ library. For instance, say we want to transmit
an integer value from Python:

  comm.send(1, 0, 17)

In C++, we would receive that value into a Python object and then
`extract` an integer value:

[c++]

  boost::python::object value;
  comm.recv(0, 0, value);
  int int_value = boost::python::extract<int>(value);

In the future, Boost.MPI will be extended to allow improved
interoperability with the C++ Boost.MPI and the C MPI bindings.
[endsect:compatibility]

[section:reference Reference]
The Boost.MPI Python module, `boost.mpi`, has its own
[@boost.mpi.html reference documentation], which is also
available using `pydoc` (from the command line) or
`help(boost.mpi)` (from the Python interpreter).

[endsect:reference]

[endsect:python]