1<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd"> 2<html> 3<head> 4<meta http-equiv="Content-Type" content="text/html; charset=UTF-8"> 5<title>Python Bindings</title> 6<link rel="stylesheet" href="../../../doc/src/boostbook.css" type="text/css"> 7<meta name="generator" content="DocBook XSL Stylesheets V1.79.1"> 8<link rel="home" href="../index.html" title="The Boost C++ Libraries BoostBook Documentation Subset"> 9<link rel="up" href="../mpi.html" title="Chapter 26. Boost.MPI"> 10<link rel="prev" href="../boost/mpi/timer.html" title="Class timer"> 11<link rel="next" href="design.html" title="Design Philosophy"> 12</head> 13<body bgcolor="white" text="black" link="#0000FF" vlink="#840084" alink="#0000FF"> 14<table cellpadding="2" width="100%"><tr> 15<td valign="top"><img alt="Boost C++ Libraries" width="277" height="86" src="../../../boost.png"></td> 16<td align="center"><a href="../../../index.html">Home</a></td> 17<td align="center"><a href="../../../libs/libraries.htm">Libraries</a></td> 18<td align="center"><a href="http://www.boost.org/users/people.html">People</a></td> 19<td align="center"><a href="http://www.boost.org/users/faq.html">FAQ</a></td> 20<td align="center"><a href="../../../more/index.htm">More</a></td> 21</tr></table> 22<hr> 23<div class="spirit-nav"> 24<a accesskey="p" href="../boost/mpi/timer.html"><img src="../../../doc/src/images/prev.png" alt="Prev"></a><a accesskey="u" href="../mpi.html"><img src="../../../doc/src/images/up.png" alt="Up"></a><a accesskey="h" href="../index.html"><img src="../../../doc/src/images/home.png" alt="Home"></a><a accesskey="n" href="design.html"><img src="../../../doc/src/images/next.png" alt="Next"></a> 25</div> 26<div class="section"> 27<div class="titlepage"><div><div><h2 class="title" style="clear: both"> 28<a name="mpi.python"></a><a class="link" href="python.html" title="Python Bindings">Python Bindings</a> 29</h2></div></div></div> 30<div class="toc"><dl class="toc"> 31<dt><span class="section"><a href="python.html#mpi.python.quickstart">Quickstart</a></span></dt> 32<dt><span class="section"><a href="python.html#mpi.python.user_data">Transmitting User-Defined Data</a></span></dt> 33<dt><span class="section"><a href="python.html#mpi.python.collectives">Collectives</a></span></dt> 34<dt><span class="section"><a href="python.html#mpi.python.skeleton_content">Skeleton/Content Mechanism</a></span></dt> 35<dt><span class="section"><a href="python.html#mpi.python.compatibility">C++/Python MPI Compatibility</a></span></dt> 36<dt><span class="section"><a href="python.html#mpi.python.reference">Reference</a></span></dt> 37</dl></div> 38<p> 39 Boost.MPI provides an alternative MPI interface from the <a href="http://www.python.org" target="_top">Python</a> 40 programming language via the <code class="computeroutput"><span class="identifier">boost</span><span class="special">.</span><span class="identifier">mpi</span></code> module. 41 The Boost.MPI Python bindings, built on top of the C++ Boost.MPI using the 42 <a href="http://www.boost.org/libs/python/doc" target="_top">Boost.Python</a> library, 43 provide nearly all of the functionality of Boost.MPI within a dynamic, object-oriented 44 language. 45 </p> 46<p> 47 The Boost.MPI Python module can be built and installed from the <code class="computeroutput"><span class="identifier">libs</span><span class="special">/</span><span class="identifier">mpi</span><span class="special">/</span><span class="identifier">build</span></code> directory. 48 Just follow the <a class="link" href="getting_started.html#mpi.getting_started.config" title="Configure and Build">configuration</a> 49 and <a class="link" href="getting_started.html#mpi.getting_started.config.installation" title="Installation">installation</a> 50 instructions for the C++ Boost.MPI. Once you have installed the Python module, 51 be sure that the installation location is in your <code class="computeroutput"><span class="identifier">PYTHONPATH</span></code>. 52 </p> 53<div class="section"> 54<div class="titlepage"><div><div><h3 class="title"> 55<a name="mpi.python.quickstart"></a><a class="link" href="python.html#mpi.python.quickstart" title="Quickstart">Quickstart</a> 56</h3></div></div></div> 57<p> 58 Getting started with the Boost.MPI Python module is as easy as importing 59 <code class="computeroutput"><span class="identifier">boost</span><span class="special">.</span><span class="identifier">mpi</span></code>. Our first "Hello, World!" 60 program is just two lines long: 61 </p> 62<pre class="programlisting"><span class="keyword">import</span> <span class="identifier">boost</span><span class="special">.</span><span class="identifier">mpi</span> <span class="keyword">as</span> <span class="identifier">mpi</span> 63<span class="keyword">print</span> <span class="string">"I am process %d of %d."</span> <span class="special">%</span> <span class="special">(</span><span class="identifier">mpi</span><span class="special">.</span><span class="identifier">rank</span><span class="special">,</span> <span class="identifier">mpi</span><span class="special">.</span><span class="identifier">size</span><span class="special">)</span> 64</pre> 65<p> 66 Go ahead and run this program with several processes. Be sure to invoke the 67 <code class="computeroutput"><span class="identifier">python</span></code> interpreter from 68 <code class="computeroutput"><span class="identifier">mpirun</span></code>, e.g., 69 </p> 70<pre class="programlisting">mpirun -np 5 python hello_world.py 71</pre> 72<p> 73 This will return output such as: 74 </p> 75<pre class="programlisting">I am process 1 of 5. 76I am process 3 of 5. 77I am process 2 of 5. 78I am process 4 of 5. 79I am process 0 of 5. 80</pre> 81<p> 82 Point-to-point operations in Boost.MPI have nearly the same syntax in Python 83 as in C++. We can write a simple two-process Python program that prints "Hello, 84 world!" by transmitting Python strings: 85 </p> 86<pre class="programlisting"><span class="keyword">import</span> <span class="identifier">boost</span><span class="special">.</span><span class="identifier">mpi</span> <span class="keyword">as</span> <span class="identifier">mpi</span> 87 88<span class="keyword">if</span> <span class="identifier">mpi</span><span class="special">.</span><span class="identifier">world</span><span class="special">.</span><span class="identifier">rank</span> <span class="special">==</span> <span class="number">0</span><span class="special">:</span> 89 <span class="identifier">mpi</span><span class="special">.</span><span class="identifier">world</span><span class="special">.</span><span class="identifier">send</span><span class="special">(</span><span class="number">1</span><span class="special">,</span> <span class="number">0</span><span class="special">,</span> <span class="string">'Hello'</span><span class="special">)</span> 90 <span class="identifier">msg</span> <span class="special">=</span> <span class="identifier">mpi</span><span class="special">.</span><span class="identifier">world</span><span class="special">.</span><span class="identifier">recv</span><span class="special">(</span><span class="number">1</span><span class="special">,</span> <span class="number">1</span><span class="special">)</span> 91 <span class="keyword">print</span> <span class="identifier">msg</span><span class="special">,</span><span class="string">'!'</span> 92<span class="keyword">else</span><span class="special">:</span> 93 <span class="identifier">msg</span> <span class="special">=</span> <span class="identifier">mpi</span><span class="special">.</span><span class="identifier">world</span><span class="special">.</span><span class="identifier">recv</span><span class="special">(</span><span class="number">0</span><span class="special">,</span> <span class="number">0</span><span class="special">)</span> 94 <span class="keyword">print</span> <span class="special">(</span><span class="identifier">msg</span> <span class="special">+</span> <span class="string">', '</span><span class="special">),</span> 95 <span class="identifier">mpi</span><span class="special">.</span><span class="identifier">world</span><span class="special">.</span><span class="identifier">send</span><span class="special">(</span><span class="number">0</span><span class="special">,</span> <span class="number">1</span><span class="special">,</span> <span class="string">'world'</span><span class="special">)</span> 96</pre> 97<p> 98 There are only a few notable differences between this Python code and the 99 example <a class="link" href="tutorial.html#mpi.tutorial.point_to_point" title="Point-to-Point communication">in the C++ tutorial</a>. 100 First of all, we don't need to write any initialization code in Python: just 101 loading the <code class="computeroutput"><span class="identifier">boost</span><span class="special">.</span><span class="identifier">mpi</span></code> module makes the appropriate <code class="computeroutput"><span class="identifier">MPI_Init</span></code> and <code class="computeroutput"><span class="identifier">MPI_Finalize</span></code> 102 calls. Second, we're passing Python objects from one process to another through 103 MPI. Any Python object that can be pickled can be transmitted; the next section 104 will describe in more detail how the Boost.MPI Python layer transmits objects. 105 Finally, when we receive objects with <code class="computeroutput"><span class="identifier">recv</span></code>, 106 we don't need to specify the type because transmission of Python objects 107 is polymorphic. 108 </p> 109<p> 110 When experimenting with Boost.MPI in Python, don't forget that help is always 111 available via <code class="computeroutput"><span class="identifier">pydoc</span></code>: just 112 pass the name of the module or module entity on the command line (e.g., 113 <code class="computeroutput"><span class="identifier">pydoc</span> <span class="identifier">boost</span><span class="special">.</span><span class="identifier">mpi</span><span class="special">.</span><span class="identifier">communicator</span></code>) to receive complete reference 114 documentation. When in doubt, try it! 115 </p> 116</div> 117<div class="section"> 118<div class="titlepage"><div><div><h3 class="title"> 119<a name="mpi.python.user_data"></a><a class="link" href="python.html#mpi.python.user_data" title="Transmitting User-Defined Data">Transmitting User-Defined Data</a> 120</h3></div></div></div> 121<p> 122 Boost.MPI can transmit user-defined data in several different ways. Most 123 importantly, it can transmit arbitrary <a href="http://www.python.org" target="_top">Python</a> 124 objects by pickling them at the sender and unpickling them at the receiver, 125 allowing arbitrarily complex Python data structures to interoperate with 126 MPI. 127 </p> 128<p> 129 Boost.MPI also supports efficient serialization and transmission of C++ objects 130 (that have been exposed to Python) through its C++ interface. Any C++ type 131 that provides (de-)serialization routines that meet the requirements of the 132 Boost.Serialization library is eligible for this optimization, but the type 133 must be registered in advance. To register a C++ type, invoke the C++ function 134 <code class="computeroutput"><a class="link" href="../boost/mpi/python/register_serialized.html" title="Function template register_serialized">register_serialized</a></code>. 135 If your C++ types come from other Python modules (they probably will!), those 136 modules will need to link against the <code class="computeroutput"><span class="identifier">boost_mpi</span></code> 137 and <code class="computeroutput"><span class="identifier">boost_mpi_python</span></code> libraries 138 as described in the <a class="link" href="getting_started.html#mpi.getting_started.config.installation" title="Installation">installation 139 section</a>. Note that you do <span class="bold"><strong>not</strong></span> need 140 to link against the Boost.MPI Python extension module. 141 </p> 142<p> 143 Finally, Boost.MPI supports separation of the structure of an object from 144 the data it stores, allowing the two pieces to be transmitted separately. 145 This "skeleton/content" mechanism, described in more detail in 146 a later section, is a communication optimization suitable for problems with 147 fixed data structures whose internal data changes frequently. 148 </p> 149</div> 150<div class="section"> 151<div class="titlepage"><div><div><h3 class="title"> 152<a name="mpi.python.collectives"></a><a class="link" href="python.html#mpi.python.collectives" title="Collectives">Collectives</a> 153</h3></div></div></div> 154<p> 155 Boost.MPI supports all of the MPI collectives (<code class="computeroutput"><span class="identifier">scatter</span></code>, 156 <code class="computeroutput"><span class="identifier">reduce</span></code>, <code class="computeroutput"><span class="identifier">scan</span></code>, 157 <code class="computeroutput"><span class="identifier">broadcast</span></code>, etc.) for any 158 type of data that can be transmitted with the point-to-point communication 159 operations. For the MPI collectives that require a user-specified operation 160 (e.g., <code class="computeroutput"><span class="identifier">reduce</span></code> and <code class="computeroutput"><span class="identifier">scan</span></code>), the operation can be an arbitrary 161 Python function. For instance, one could concatenate strings with <code class="computeroutput"><span class="identifier">all_reduce</span></code>: 162 </p> 163<pre class="programlisting"><span class="identifier">mpi</span><span class="special">.</span><span class="identifier">all_reduce</span><span class="special">(</span><span class="identifier">my_string</span><span class="special">,</span> <span class="keyword">lambda</span> <span class="identifier">x</span><span class="special">,</span><span class="identifier">y</span><span class="special">:</span> <span class="identifier">x</span> <span class="special">+</span> <span class="identifier">y</span><span class="special">)</span> 164</pre> 165<p> 166 The following module-level functions implement MPI collectives: all_gather 167 Gather the values from all processes. all_reduce Combine the results from 168 all processes. all_to_all Every process sends data to every other process. 169 broadcast Broadcast data from one process to all other processes. gather 170 Gather the values from all processes to the root. reduce Combine the results 171 from all processes to the root. scan Prefix reduction of the values from 172 all processes. scatter Scatter the values stored at the root to all processes. 173 </p> 174</div> 175<div class="section"> 176<div class="titlepage"><div><div><h3 class="title"> 177<a name="mpi.python.skeleton_content"></a><a class="link" href="python.html#mpi.python.skeleton_content" title="Skeleton/Content Mechanism">Skeleton/Content Mechanism</a> 178</h3></div></div></div> 179<p> 180 Boost.MPI provides a skeleton/content mechanism that allows the transfer 181 of large data structures to be split into two separate stages, with the skeleton 182 (or, "shape") of the data structure sent first and the content 183 (or, "data") of the data structure sent later, potentially several 184 times, so long as the structure has not changed since the skeleton was transferred. 185 The skeleton/content mechanism can improve performance when the data structure 186 is large and its shape is fixed, because while the skeleton requires serialization 187 (it has an unknown size), the content transfer is fixed-size and can be done 188 without extra copies. 189 </p> 190<p> 191 To use the skeleton/content mechanism from Python, you must first register 192 the type of your data structure with the skeleton/content mechanism <span class="bold"><strong>from C++</strong></span>. The registration function is <code class="computeroutput"><a class="link" href="../boost/mpi/python/regist_1_3_27_7_20_2_1_1_2.html" title="Function template register_skeleton_and_content">register_skeleton_and_content</a></code> 193 and resides in the <code class="computeroutput"><a class="link" href="reference.html#header.boost.mpi.python_hpp" title="Header <boost/mpi/python.hpp>"><boost/mpi/python.hpp></a></code> 194 header. 195 </p> 196<p> 197 Once you have registered your C++ data structures, you can extract the skeleton 198 for an instance of that data structure with <code class="computeroutput"><span class="identifier">skeleton</span><span class="special">()</span></code>. The resulting <code class="computeroutput"><span class="identifier">skeleton_proxy</span></code> 199 can be transmitted via the normal send routine, e.g., 200 </p> 201<pre class="programlisting"><span class="identifier">mpi</span><span class="special">.</span><span class="identifier">world</span><span class="special">.</span><span class="identifier">send</span><span class="special">(</span><span class="number">1</span><span class="special">,</span> <span class="number">0</span><span class="special">,</span> <span class="identifier">skeleton</span><span class="special">(</span><span class="identifier">my_data_structure</span><span class="special">))</span> 202</pre> 203<p> 204 <code class="computeroutput"><span class="identifier">skeleton_proxy</span></code> objects can 205 be received on the other end via <code class="computeroutput"><span class="identifier">recv</span><span class="special">()</span></code>, which stores a newly-created instance 206 of your data structure with the same "shape" as the sender in its 207 <code class="computeroutput"><span class="string">"object"</span></code> attribute: 208 </p> 209<pre class="programlisting"><span class="identifier">shape</span> <span class="special">=</span> <span class="identifier">mpi</span><span class="special">.</span><span class="identifier">world</span><span class="special">.</span><span class="identifier">recv</span><span class="special">(</span><span class="number">0</span><span class="special">,</span> <span class="number">0</span><span class="special">)</span> 210<span class="identifier">my_data_structure</span> <span class="special">=</span> <span class="identifier">shape</span><span class="special">.</span><span class="identifier">object</span> 211</pre> 212<p> 213 Once the skeleton has been transmitted, the content (accessed via <code class="computeroutput"><span class="identifier">get_content</span></code>) can be transmitted in much 214 the same way. Note, however, that the receiver also specifies <code class="computeroutput"><span class="identifier">get_content</span><span class="special">(</span><span class="identifier">my_data_structure</span><span class="special">)</span></code> 215 in its call to receive: 216 </p> 217<pre class="programlisting"><span class="keyword">if</span> <span class="identifier">mpi</span><span class="special">.</span><span class="identifier">rank</span> <span class="special">==</span> <span class="number">0</span><span class="special">:</span> 218 <span class="identifier">mpi</span><span class="special">.</span><span class="identifier">world</span><span class="special">.</span><span class="identifier">send</span><span class="special">(</span><span class="number">1</span><span class="special">,</span> <span class="number">0</span><span class="special">,</span> <span class="identifier">get_content</span><span class="special">(</span><span class="identifier">my_data_structure</span><span class="special">))</span> 219<span class="keyword">else</span><span class="special">:</span> 220 <span class="identifier">mpi</span><span class="special">.</span><span class="identifier">world</span><span class="special">.</span><span class="identifier">recv</span><span class="special">(</span><span class="number">0</span><span class="special">,</span> <span class="number">0</span><span class="special">,</span> <span class="identifier">get_content</span><span class="special">(</span><span class="identifier">my_data_structure</span><span class="special">))</span> 221</pre> 222<p> 223 Of course, this transmission of content can occur repeatedly, if the values 224 in the data structure--but not its shape--changes. 225 </p> 226<p> 227 The skeleton/content mechanism is a structured way to exploit the interaction 228 between custom-built MPI datatypes and <code class="computeroutput"><span class="identifier">MPI_BOTTOM</span></code>, 229 to eliminate extra buffer copies. 230 </p> 231</div> 232<div class="section"> 233<div class="titlepage"><div><div><h3 class="title"> 234<a name="mpi.python.compatibility"></a><a class="link" href="python.html#mpi.python.compatibility" title="C++/Python MPI Compatibility">C++/Python MPI Compatibility</a> 235</h3></div></div></div> 236<p> 237 Boost.MPI is a C++ library whose facilities have been exposed to Python via 238 the Boost.Python library. Since the Boost.MPI Python bindings are build directly 239 on top of the C++ library, and nearly every feature of C++ library is available 240 in Python, hybrid C++/Python programs using Boost.MPI can interact, e.g., 241 sending a value from Python but receiving that value in C++ (or vice versa). 242 However, doing so requires some care. Because Python objects are dynamically 243 typed, Boost.MPI transfers type information along with the serialized form 244 of the object, so that the object can be received even when its type is not 245 known. This mechanism differs from its C++ counterpart, where the static 246 types of transmitted values are always known. 247 </p> 248<p> 249 The only way to communicate between the C++ and Python views on Boost.MPI 250 is to traffic entirely in Python objects. For Python, this is the normal 251 state of affairs, so nothing will change. For C++, this means sending and 252 receiving values of type <code class="computeroutput"><span class="identifier">boost</span><span class="special">::</span><span class="identifier">python</span><span class="special">::</span><span class="identifier">object</span></code>, 253 from the <a href="http://www.boost.org/libs/python/doc" target="_top">Boost.Python</a> 254 library. For instance, say we want to transmit an integer value from Python: 255 </p> 256<pre class="programlisting"><span class="identifier">comm</span><span class="special">.</span><span class="identifier">send</span><span class="special">(</span><span class="number">1</span><span class="special">,</span> <span class="number">0</span><span class="special">,</span> <span class="number">17</span><span class="special">)</span> 257</pre> 258<p> 259 In C++, we would receive that value into a Python object and then <code class="computeroutput"><span class="identifier">extract</span></code> an integer value: 260 </p> 261<pre class="programlisting"><span class="identifier">boost</span><span class="special">::</span><span class="identifier">python</span><span class="special">::</span><span class="identifier">object</span> <span class="identifier">value</span><span class="special">;</span> 262<span class="identifier">comm</span><span class="special">.</span><span class="identifier">recv</span><span class="special">(</span><span class="number">0</span><span class="special">,</span> <span class="number">0</span><span class="special">,</span> <span class="identifier">value</span><span class="special">);</span> 263<span class="keyword">int</span> <span class="identifier">int_value</span> <span class="special">=</span> <span class="identifier">boost</span><span class="special">::</span><span class="identifier">python</span><span class="special">::</span><span class="identifier">extract</span><span class="special"><</span><span class="keyword">int</span><span class="special">>(</span><span class="identifier">value</span><span class="special">);</span> 264</pre> 265<p> 266 In the future, Boost.MPI will be extended to allow improved interoperability 267 with the C++ Boost.MPI and the C MPI bindings. 268 </p> 269</div> 270<div class="section"> 271<div class="titlepage"><div><div><h3 class="title"> 272<a name="mpi.python.reference"></a><a class="link" href="python.html#mpi.python.reference" title="Reference">Reference</a> 273</h3></div></div></div> 274<p> 275 The Boost.MPI Python module, <code class="computeroutput"><span class="identifier">boost</span><span class="special">.</span><span class="identifier">mpi</span></code>, has 276 its own <a href="../boost.mpi.html" target="_top">reference documentation</a>, which 277 is also available using <code class="computeroutput"><span class="identifier">pydoc</span></code> 278 (from the command line) or <code class="computeroutput"><span class="identifier">help</span><span class="special">(</span><span class="identifier">boost</span><span class="special">.</span><span class="identifier">mpi</span><span class="special">)</span></code> 279 (from the Python interpreter). 280 </p> 281</div> 282</div> 283<table xmlns:rev="http://www.cs.rpi.edu/~gregod/boost/tools/doc/revision" width="100%"><tr> 284<td align="left"></td> 285<td align="right"><div class="copyright-footer">Copyright © 2005-2007 Douglas Gregor, 286 Matthias Troyer, Trustees of Indiana University<p> 287 Distributed under the Boost Software License, Version 1.0. (See accompanying 288 file LICENSE_1_0.txt or copy at <ulink url="http://www.boost.org/LICENSE_1_0.txt"> 289 http://www.boost.org/LICENSE_1_0.txt </ulink>) 290 </p> 291</div></td> 292</tr></table> 293<hr> 294<div class="spirit-nav"> 295<a accesskey="p" href="../boost/mpi/timer.html"><img src="../../../doc/src/images/prev.png" alt="Prev"></a><a accesskey="u" href="../mpi.html"><img src="../../../doc/src/images/up.png" alt="Up"></a><a accesskey="h" href="../index.html"><img src="../../../doc/src/images/home.png" alt="Home"></a><a accesskey="n" href="design.html"><img src="../../../doc/src/images/next.png" alt="Next"></a> 296</div> 297</body> 298</html> 299
