1<html> 2<head> 3<meta http-equiv="Content-Type" content="text/html; charset=UTF-8"> 4<title>Parallel computation with OpenMP and MPI</title> 5<link rel="stylesheet" href="../../../../../../../doc/src/boostbook.css" type="text/css"> 6<meta name="generator" content="DocBook XSL Stylesheets V1.79.1"> 7<link rel="home" href="../../index.html" title="Chapter 1. Boost.Numeric.Odeint"> 8<link rel="up" href="../tutorial.html" title="Tutorial"> 9<link rel="prev" href="using_opencl_via_vexcl.html" title="Using OpenCL via VexCL"> 10<link rel="next" href="all_examples.html" title="All examples"> 11</head> 12<body bgcolor="white" text="black" link="#0000FF" vlink="#840084" alink="#0000FF"> 13<table cellpadding="2" width="100%"><tr> 14<td valign="top"><img alt="Boost C++ Libraries" width="277" height="86" src="../../logo.jpg"></td> 15<td align="center"><a href="../../../../../../../index.html">Home</a></td> 16<td align="center"><a href="../../../../../../../libs/libraries.htm">Libraries</a></td> 17<td align="center"><a href="http://www.boost.org/users/people.html">People</a></td> 18<td align="center"><a href="http://www.boost.org/users/faq.html">FAQ</a></td> 19<td align="center"><a href="../../../../../../../more/index.htm">More</a></td> 20</tr></table> 21<hr> 22<div class="spirit-nav"> 23<a accesskey="p" href="using_opencl_via_vexcl.html"><img src="../../../../../../../doc/src/images/prev.png" alt="Prev"></a><a accesskey="u" href="../tutorial.html"><img src="../../../../../../../doc/src/images/up.png" alt="Up"></a><a accesskey="h" href="../../index.html"><img src="../../../../../../../doc/src/images/home.png" alt="Home"></a><a accesskey="n" href="all_examples.html"><img src="../../../../../../../doc/src/images/next.png" alt="Next"></a> 24</div> 25<div class="section"> 26<div class="titlepage"><div><div><h3 class="title"> 27<a name="boost_numeric_odeint.tutorial.parallel_computation_with_openmp_and_mpi"></a><a class="link" href="parallel_computation_with_openmp_and_mpi.html" title="Parallel computation with OpenMP and MPI">Parallel 28 computation with OpenMP and MPI</a> 29</h3></div></div></div> 30<div class="toc"><dl class="toc"> 31<dt><span class="section"><a href="parallel_computation_with_openmp_and_mpi.html#boost_numeric_odeint.tutorial.parallel_computation_with_openmp_and_mpi.openmp">OpenMP</a></span></dt> 32<dt><span class="section"><a href="parallel_computation_with_openmp_and_mpi.html#boost_numeric_odeint.tutorial.parallel_computation_with_openmp_and_mpi.mpi">MPI</a></span></dt> 33<dt><span class="section"><a href="parallel_computation_with_openmp_and_mpi.html#boost_numeric_odeint.tutorial.parallel_computation_with_openmp_and_mpi.concepts">Concepts</a></span></dt> 34</dl></div> 35<p> 36 Parallelization is a key feature for modern numerical libraries due to the 37 vast availability of many cores nowadays, even on Laptops. odeint currently 38 supports parallelization with OpenMP and MPI, as described in the following 39 sections. However, it should be made clear from the beginning that the difficulty 40 of efficiently distributing ODE integration on many cores/machines lies in 41 the parallelization of the system function, which is still the user's responsibility. 42 Simply using a parallel odeint backend without parallelizing the system function 43 will bring you almost no performance gains. 44 </p> 45<div class="section"> 46<div class="titlepage"><div><div><h4 class="title"> 47<a name="boost_numeric_odeint.tutorial.parallel_computation_with_openmp_and_mpi.openmp"></a><a class="link" href="parallel_computation_with_openmp_and_mpi.html#boost_numeric_odeint.tutorial.parallel_computation_with_openmp_and_mpi.openmp" title="OpenMP">OpenMP</a> 48</h4></div></div></div> 49<p> 50 odeint's OpenMP support is implemented as an external backend, which needs 51 to be manually included. Depending on the compiler some additional flags 52 may be needed, i.e. <code class="literal">-fopenmp</code> for GCC. 53 </p> 54<p> 55</p> 56<pre class="programlisting"><span class="preprocessor">#include</span> <span class="special"><</span><span class="identifier">omp</span><span class="special">.</span><span class="identifier">h</span><span class="special">></span> 57<span class="preprocessor">#include</span> <span class="special"><</span><span class="identifier">boost</span><span class="special">/</span><span class="identifier">numeric</span><span class="special">/</span><span class="identifier">odeint</span><span class="special">.</span><span class="identifier">hpp</span><span class="special">></span> 58<span class="preprocessor">#include</span> <span class="special"><</span><span class="identifier">boost</span><span class="special">/</span><span class="identifier">numeric</span><span class="special">/</span><span class="identifier">odeint</span><span class="special">/</span><span class="identifier">external</span><span class="special">/</span><span class="identifier">openmp</span><span class="special">/</span><span class="identifier">openmp</span><span class="special">.</span><span class="identifier">hpp</span><span class="special">></span> 59</pre> 60<p> 61 </p> 62<p> 63 In the easiest parallelization approach with OpenMP we use a standard 64 <code class="computeroutput"><span class="identifier">vector</span></code> as the state type: 65 </p> 66<p> 67</p> 68<pre class="programlisting"><span class="keyword">typedef</span> <span class="identifier">std</span><span class="special">::</span><span class="identifier">vector</span><span class="special"><</span> <span class="keyword">double</span> <span class="special">></span> <span class="identifier">state_type</span><span class="special">;</span> 69</pre> 70<p> 71 </p> 72<p> 73 We initialize the state with some random data: 74 </p> 75<p> 76</p> 77<pre class="programlisting"><span class="identifier">size_t</span> <span class="identifier">N</span> <span class="special">=</span> <span class="number">131101</span><span class="special">;</span> 78<span class="identifier">state_type</span> <span class="identifier">x</span><span class="special">(</span> <span class="identifier">N</span> <span class="special">);</span> 79<span class="identifier">boost</span><span class="special">::</span><span class="identifier">random</span><span class="special">::</span><span class="identifier">uniform_real_distribution</span><span class="special"><</span><span class="keyword">double</span><span class="special">></span> <span class="identifier">distribution</span><span class="special">(</span> <span class="number">0.0</span> <span class="special">,</span> <span class="number">2.0</span><span class="special">*</span><span class="identifier">pi</span> <span class="special">);</span> 80<span class="identifier">boost</span><span class="special">::</span><span class="identifier">random</span><span class="special">::</span><span class="identifier">mt19937</span> <span class="identifier">engine</span><span class="special">(</span> <span class="number">0</span> <span class="special">);</span> 81<span class="identifier">generate</span><span class="special">(</span> <span class="identifier">x</span><span class="special">.</span><span class="identifier">begin</span><span class="special">()</span> <span class="special">,</span> <span class="identifier">x</span><span class="special">.</span><span class="identifier">end</span><span class="special">()</span> <span class="special">,</span> <span class="identifier">boost</span><span class="special">::</span><span class="identifier">bind</span><span class="special">(</span> <span class="identifier">distribution</span> <span class="special">,</span> <span class="identifier">engine</span> <span class="special">)</span> <span class="special">);</span> 82</pre> 83<p> 84 </p> 85<p> 86 Now we have to configure the stepper to use the OpenMP backend. This is 87 done by explicitly providing the <code class="computeroutput"><span class="identifier">openmp_range_algebra</span></code> 88 as a template parameter to the stepper. This algebra requires the state 89 type to be a model of Random Access Range and will be used from multiple 90 threads by the algebra. 91 </p> 92<p> 93</p> 94<pre class="programlisting"><span class="keyword">typedef</span> <span class="identifier">runge_kutta4</span><span class="special"><</span> 95 <span class="identifier">state_type</span> <span class="special">,</span> <span class="keyword">double</span> <span class="special">,</span> 96 <span class="identifier">state_type</span> <span class="special">,</span> <span class="keyword">double</span> <span class="special">,</span> 97 <span class="identifier">openmp_range_algebra</span> 98 <span class="special">></span> <span class="identifier">stepper_type</span><span class="special">;</span> 99</pre> 100<p> 101 </p> 102<p> 103 Additional to providing the stepper with OpenMP parallelization we also 104 need a parallelized system function to exploit the available cores. Here 105 this is shown for a simple one-dimensional chain of phase oscillators with 106 nearest neighbor coupling: 107 </p> 108<p> 109</p> 110<pre class="programlisting"><span class="keyword">struct</span> <span class="identifier">phase_chain</span> 111<span class="special">{</span> 112 <span class="identifier">phase_chain</span><span class="special">(</span> <span class="keyword">double</span> <span class="identifier">gamma</span> <span class="special">=</span> <span class="number">0.5</span> <span class="special">)</span> 113 <span class="special">:</span> <span class="identifier">m_gamma</span><span class="special">(</span> <span class="identifier">gamma</span> <span class="special">)</span> <span class="special">{</span> <span class="special">}</span> 114 115 <span class="keyword">void</span> <span class="keyword">operator</span><span class="special">()(</span> <span class="keyword">const</span> <span class="identifier">state_type</span> <span class="special">&</span><span class="identifier">x</span> <span class="special">,</span> <span class="identifier">state_type</span> <span class="special">&</span><span class="identifier">dxdt</span> <span class="special">,</span> <span class="keyword">double</span> <span class="comment">/* t */</span> <span class="special">)</span> <span class="keyword">const</span> 116 <span class="special">{</span> 117 <span class="keyword">const</span> <span class="identifier">size_t</span> <span class="identifier">N</span> <span class="special">=</span> <span class="identifier">x</span><span class="special">.</span><span class="identifier">size</span><span class="special">();</span> 118 <span class="preprocessor">#pragma</span> <span class="identifier">omp</span> <span class="identifier">parallel</span> <span class="keyword">for</span> <span class="identifier">schedule</span><span class="special">(</span><span class="identifier">runtime</span><span class="special">)</span> 119 <span class="keyword">for</span><span class="special">(</span><span class="identifier">size_t</span> <span class="identifier">i</span> <span class="special">=</span> <span class="number">1</span> <span class="special">;</span> <span class="identifier">i</span> <span class="special"><</span> <span class="identifier">N</span> <span class="special">-</span> <span class="number">1</span> <span class="special">;</span> <span class="special">++</span><span class="identifier">i</span><span class="special">)</span> 120 <span class="special">{</span> 121 <span class="identifier">dxdt</span><span class="special">[</span><span class="identifier">i</span><span class="special">]</span> <span class="special">=</span> <span class="identifier">coupling_func</span><span class="special">(</span> <span class="identifier">x</span><span class="special">[</span><span class="identifier">i</span><span class="special">+</span><span class="number">1</span><span class="special">]</span> <span class="special">-</span> <span class="identifier">x</span><span class="special">[</span><span class="identifier">i</span><span class="special">]</span> <span class="special">)</span> <span class="special">+</span> 122 <span class="identifier">coupling_func</span><span class="special">(</span> <span class="identifier">x</span><span class="special">[</span><span class="identifier">i</span><span class="special">-</span><span class="number">1</span><span class="special">]</span> <span class="special">-</span> <span class="identifier">x</span><span class="special">[</span><span class="identifier">i</span><span class="special">]</span> <span class="special">);</span> 123 <span class="special">}</span> 124 <span class="identifier">dxdt</span><span class="special">[</span><span class="number">0</span> <span class="special">]</span> <span class="special">=</span> <span class="identifier">coupling_func</span><span class="special">(</span> <span class="identifier">x</span><span class="special">[</span><span class="number">1</span> <span class="special">]</span> <span class="special">-</span> <span class="identifier">x</span><span class="special">[</span><span class="number">0</span> <span class="special">]</span> <span class="special">);</span> 125 <span class="identifier">dxdt</span><span class="special">[</span><span class="identifier">N</span><span class="special">-</span><span class="number">1</span><span class="special">]</span> <span class="special">=</span> <span class="identifier">coupling_func</span><span class="special">(</span> <span class="identifier">x</span><span class="special">[</span><span class="identifier">N</span><span class="special">-</span><span class="number">2</span><span class="special">]</span> <span class="special">-</span> <span class="identifier">x</span><span class="special">[</span><span class="identifier">N</span><span class="special">-</span><span class="number">1</span><span class="special">]</span> <span class="special">);</span> 126 <span class="special">}</span> 127 128 <span class="keyword">double</span> <span class="identifier">coupling_func</span><span class="special">(</span> <span class="keyword">double</span> <span class="identifier">x</span> <span class="special">)</span> <span class="keyword">const</span> 129 <span class="special">{</span> 130 <span class="keyword">return</span> <span class="identifier">sin</span><span class="special">(</span> <span class="identifier">x</span> <span class="special">)</span> <span class="special">-</span> <span class="identifier">m_gamma</span> <span class="special">*</span> <span class="special">(</span> <span class="number">1.0</span> <span class="special">-</span> <span class="identifier">cos</span><span class="special">(</span> <span class="identifier">x</span> <span class="special">)</span> <span class="special">);</span> 131 <span class="special">}</span> 132 133 <span class="keyword">double</span> <span class="identifier">m_gamma</span><span class="special">;</span> 134<span class="special">};</span> 135</pre> 136<p> 137 </p> 138<div class="note"><table border="0" summary="Note"> 139<tr> 140<td rowspan="2" align="center" valign="top" width="25"><img alt="[Note]" src="../../../../../../../doc/src/images/note.png"></td> 141<th align="left">Note</th> 142</tr> 143<tr><td align="left" valign="top"><p> 144 In the OpenMP backends the system function will always be called sequentially 145 from the thread used to start the integration. 146 </p></td></tr> 147</table></div> 148<p> 149 Finally, we perform the integration by using one of the integrate functions 150 from odeint. As you can see, the parallelization is completely hidden in 151 the stepper and the system function. OpenMP will take care of distributing 152 the work among the threads and join them automatically. 153 </p> 154<p> 155</p> 156<pre class="programlisting"><span class="identifier">integrate_n_steps</span><span class="special">(</span> <span class="identifier">stepper_type</span><span class="special">()</span> <span class="special">,</span> <span class="identifier">phase_chain</span><span class="special">(</span> <span class="number">1.2</span> <span class="special">)</span> <span class="special">,</span> 157 <span class="identifier">x</span> <span class="special">,</span> <span class="number">0.0</span> <span class="special">,</span> <span class="number">0.01</span> <span class="special">,</span> <span class="number">100</span> <span class="special">);</span> 158</pre> 159<p> 160 </p> 161<p> 162 After integrating, the data can be accessed immediately and be processed 163 further. Note, that you can specify the OpenMP scheduling by calling <code class="computeroutput"><span class="identifier">omp_set_schedule</span></code> in the beginning of 164 your program: 165 </p> 166<p> 167</p> 168<pre class="programlisting"><span class="keyword">int</span> <span class="identifier">chunk_size</span> <span class="special">=</span> <span class="identifier">N</span><span class="special">/</span><span class="identifier">omp_get_max_threads</span><span class="special">();</span> 169<span class="identifier">omp_set_schedule</span><span class="special">(</span> <span class="identifier">omp_sched_static</span> <span class="special">,</span> <span class="identifier">chunk_size</span> <span class="special">);</span> 170</pre> 171<p> 172 </p> 173<p> 174 See <a href="https://github.com/headmyshoulder/odeint-v2/blob/master/examples/openmp/phase_chain.cpp" target="_top">openmp/phase_chain.cpp</a> for the complete 175 example. 176 </p> 177<h6> 178<a name="boost_numeric_odeint.tutorial.parallel_computation_with_openmp_and_mpi.openmp.h0"></a> 179 <span class="phrase"><a name="boost_numeric_odeint.tutorial.parallel_computation_with_openmp_and_mpi.openmp.split_state"></a></span><a class="link" href="parallel_computation_with_openmp_and_mpi.html#boost_numeric_odeint.tutorial.parallel_computation_with_openmp_and_mpi.openmp.split_state">Split 180 state</a> 181 </h6> 182<p> 183 For advanced cases odeint offers another approach to use OpenMP that allows 184 for a more exact control of the parallelization. For example, for odd-sized 185 data where OpenMP's thread boundaries don't match cache lines and hurt 186 performance it might be advisable to copy the data from the continuous 187 <code class="computeroutput"><span class="identifier">vector</span><span class="special"><</span><span class="identifier">T</span><span class="special">></span></code> 188 into separate, individually aligned, vectors. For this, odeint provides 189 the <code class="computeroutput"><span class="identifier">openmp_state</span><span class="special"><</span><span class="identifier">T</span><span class="special">></span></code> 190 type, essentially an alias for <code class="computeroutput"><span class="identifier">vector</span><span class="special"><</span><span class="identifier">vector</span><span class="special"><</span><span class="identifier">T</span><span class="special">>></span></code>. 191 </p> 192<p> 193 Here, the initialization is done with a <code class="computeroutput"><span class="identifier">vector</span><span class="special"><</span><span class="keyword">double</span><span class="special">></span></code>, but then we use odeint's <code class="computeroutput"><span class="identifier">split</span></code> function to fill an <code class="computeroutput"><span class="identifier">openmp_state</span></code>. The splitting is done such 194 that the sizes of the individual regions differ at most by 1 to make the 195 computation as uniform as possible. 196 </p> 197<p> 198</p> 199<pre class="programlisting"><span class="keyword">const</span> <span class="identifier">size_t</span> <span class="identifier">N</span> <span class="special">=</span> <span class="number">131101</span><span class="special">;</span> 200<span class="identifier">vector</span><span class="special"><</span><span class="keyword">double</span><span class="special">></span> <span class="identifier">x</span><span class="special">(</span> <span class="identifier">N</span> <span class="special">);</span> 201<span class="identifier">boost</span><span class="special">::</span><span class="identifier">random</span><span class="special">::</span><span class="identifier">uniform_real_distribution</span><span class="special"><</span><span class="keyword">double</span><span class="special">></span> <span class="identifier">distribution</span><span class="special">(</span> <span class="number">0.0</span> <span class="special">,</span> <span class="number">2.0</span><span class="special">*</span><span class="identifier">pi</span> <span class="special">);</span> 202<span class="identifier">boost</span><span class="special">::</span><span class="identifier">random</span><span class="special">::</span><span class="identifier">mt19937</span> <span class="identifier">engine</span><span class="special">(</span> <span class="number">0</span> <span class="special">);</span> 203<span class="identifier">generate</span><span class="special">(</span> <span class="identifier">x</span><span class="special">.</span><span class="identifier">begin</span><span class="special">()</span> <span class="special">,</span> <span class="identifier">x</span><span class="special">.</span><span class="identifier">end</span><span class="special">()</span> <span class="special">,</span> <span class="identifier">boost</span><span class="special">::</span><span class="identifier">bind</span><span class="special">(</span> <span class="identifier">distribution</span> <span class="special">,</span> <span class="identifier">engine</span> <span class="special">)</span> <span class="special">);</span> 204<span class="keyword">const</span> <span class="identifier">size_t</span> <span class="identifier">blocks</span> <span class="special">=</span> <span class="identifier">omp_get_max_threads</span><span class="special">();</span> 205<span class="identifier">state_type</span> <span class="identifier">x_split</span><span class="special">(</span> <span class="identifier">blocks</span> <span class="special">);</span> 206<span class="identifier">split</span><span class="special">(</span> <span class="identifier">x</span> <span class="special">,</span> <span class="identifier">x_split</span> <span class="special">);</span> 207</pre> 208<p> 209 </p> 210<p> 211 Of course, the system function has to be changed to deal with the <code class="computeroutput"><span class="identifier">openmp_state</span></code>. Note that each sub-region 212 of the state is computed in a single task, but at the borders read access 213 to the neighbouring regions is required. 214 </p> 215<p> 216</p> 217<pre class="programlisting"><span class="keyword">struct</span> <span class="identifier">phase_chain_omp_state</span> 218<span class="special">{</span> 219 <span class="identifier">phase_chain_omp_state</span><span class="special">(</span> <span class="keyword">double</span> <span class="identifier">gamma</span> <span class="special">=</span> <span class="number">0.5</span> <span class="special">)</span> 220 <span class="special">:</span> <span class="identifier">m_gamma</span><span class="special">(</span> <span class="identifier">gamma</span> <span class="special">)</span> <span class="special">{</span> <span class="special">}</span> 221 222 <span class="keyword">void</span> <span class="keyword">operator</span><span class="special">()(</span> <span class="keyword">const</span> <span class="identifier">state_type</span> <span class="special">&</span><span class="identifier">x</span> <span class="special">,</span> <span class="identifier">state_type</span> <span class="special">&</span><span class="identifier">dxdt</span> <span class="special">,</span> <span class="keyword">double</span> <span class="comment">/* t */</span> <span class="special">)</span> <span class="keyword">const</span> 223 <span class="special">{</span> 224 <span class="keyword">const</span> <span class="identifier">size_t</span> <span class="identifier">N</span> <span class="special">=</span> <span class="identifier">x</span><span class="special">.</span><span class="identifier">size</span><span class="special">();</span> 225 <span class="preprocessor">#pragma</span> <span class="identifier">omp</span> <span class="identifier">parallel</span> <span class="keyword">for</span> <span class="identifier">schedule</span><span class="special">(</span><span class="identifier">runtime</span><span class="special">)</span> 226 <span class="keyword">for</span><span class="special">(</span><span class="identifier">size_t</span> <span class="identifier">n</span> <span class="special">=</span> <span class="number">0</span> <span class="special">;</span> <span class="identifier">n</span> <span class="special"><</span> <span class="identifier">N</span> <span class="special">;</span> <span class="special">++</span><span class="identifier">n</span><span class="special">)</span> 227 <span class="special">{</span> 228 <span class="keyword">const</span> <span class="identifier">size_t</span> <span class="identifier">M</span> <span class="special">=</span> <span class="identifier">x</span><span class="special">[</span><span class="identifier">n</span><span class="special">].</span><span class="identifier">size</span><span class="special">();</span> 229 <span class="keyword">for</span><span class="special">(</span><span class="identifier">size_t</span> <span class="identifier">m</span> <span class="special">=</span> <span class="number">1</span> <span class="special">;</span> <span class="identifier">m</span> <span class="special"><</span> <span class="identifier">M</span><span class="special">-</span><span class="number">1</span> <span class="special">;</span> <span class="special">++</span><span class="identifier">m</span><span class="special">)</span> 230 <span class="special">{</span> 231 <span class="identifier">dxdt</span><span class="special">[</span><span class="identifier">n</span><span class="special">][</span><span class="identifier">m</span><span class="special">]</span> <span class="special">=</span> <span class="identifier">coupling_func</span><span class="special">(</span> <span class="identifier">x</span><span class="special">[</span><span class="identifier">n</span><span class="special">][</span><span class="identifier">m</span><span class="special">+</span><span class="number">1</span><span class="special">]</span> <span class="special">-</span> <span class="identifier">x</span><span class="special">[</span><span class="identifier">n</span><span class="special">][</span><span class="identifier">m</span><span class="special">]</span> <span class="special">)</span> <span class="special">+</span> 232 <span class="identifier">coupling_func</span><span class="special">(</span> <span class="identifier">x</span><span class="special">[</span><span class="identifier">n</span><span class="special">][</span><span class="identifier">m</span><span class="special">-</span><span class="number">1</span><span class="special">]</span> <span class="special">-</span> <span class="identifier">x</span><span class="special">[</span><span class="identifier">n</span><span class="special">][</span><span class="identifier">m</span><span class="special">]</span> <span class="special">);</span> 233 <span class="special">}</span> 234 <span class="identifier">dxdt</span><span class="special">[</span><span class="identifier">n</span><span class="special">][</span><span class="number">0</span><span class="special">]</span> <span class="special">=</span> <span class="identifier">coupling_func</span><span class="special">(</span> <span class="identifier">x</span><span class="special">[</span><span class="identifier">n</span><span class="special">][</span><span class="number">1</span><span class="special">]</span> <span class="special">-</span> <span class="identifier">x</span><span class="special">[</span><span class="identifier">n</span><span class="special">][</span><span class="number">0</span><span class="special">]</span> <span class="special">);</span> 235 <span class="keyword">if</span><span class="special">(</span> <span class="identifier">n</span> <span class="special">></span> <span class="number">0</span> <span class="special">)</span> 236 <span class="special">{</span> 237 <span class="identifier">dxdt</span><span class="special">[</span><span class="identifier">n</span><span class="special">][</span><span class="number">0</span><span class="special">]</span> <span class="special">+=</span> <span class="identifier">coupling_func</span><span class="special">(</span> <span class="identifier">x</span><span class="special">[</span><span class="identifier">n</span><span class="special">-</span><span class="number">1</span><span class="special">].</span><span class="identifier">back</span><span class="special">()</span> <span class="special">-</span> <span class="identifier">x</span><span class="special">[</span><span class="identifier">n</span><span class="special">].</span><span class="identifier">front</span><span class="special">()</span> <span class="special">);</span> 238 <span class="special">}</span> 239 <span class="identifier">dxdt</span><span class="special">[</span><span class="identifier">n</span><span class="special">][</span><span class="identifier">M</span><span class="special">-</span><span class="number">1</span><span class="special">]</span> <span class="special">=</span> <span class="identifier">coupling_func</span><span class="special">(</span> <span class="identifier">x</span><span class="special">[</span><span class="identifier">n</span><span class="special">][</span><span class="identifier">M</span><span class="special">-</span><span class="number">2</span><span class="special">]</span> <span class="special">-</span> <span class="identifier">x</span><span class="special">[</span><span class="identifier">n</span><span class="special">][</span><span class="identifier">M</span><span class="special">-</span><span class="number">1</span><span class="special">]</span> <span class="special">);</span> 240 <span class="keyword">if</span><span class="special">(</span> <span class="identifier">n</span> <span class="special"><</span> <span class="identifier">N</span><span class="special">-</span><span class="number">1</span> <span class="special">)</span> 241 <span class="special">{</span> 242 <span class="identifier">dxdt</span><span class="special">[</span><span class="identifier">n</span><span class="special">][</span><span class="identifier">M</span><span class="special">-</span><span class="number">1</span><span class="special">]</span> <span class="special">+=</span> <span class="identifier">coupling_func</span><span class="special">(</span> <span class="identifier">x</span><span class="special">[</span><span class="identifier">n</span><span class="special">+</span><span class="number">1</span><span class="special">].</span><span class="identifier">front</span><span class="special">()</span> <span class="special">-</span> <span class="identifier">x</span><span class="special">[</span><span class="identifier">n</span><span class="special">].</span><span class="identifier">back</span><span class="special">()</span> <span class="special">);</span> 243 <span class="special">}</span> 244 <span class="special">}</span> 245 <span class="special">}</span> 246 247 <span class="keyword">double</span> <span class="identifier">coupling_func</span><span class="special">(</span> <span class="keyword">double</span> <span class="identifier">x</span> <span class="special">)</span> <span class="keyword">const</span> 248 <span class="special">{</span> 249 <span class="keyword">return</span> <span class="identifier">sin</span><span class="special">(</span> <span class="identifier">x</span> <span class="special">)</span> <span class="special">-</span> <span class="identifier">m_gamma</span> <span class="special">*</span> <span class="special">(</span> <span class="number">1.0</span> <span class="special">-</span> <span class="identifier">cos</span><span class="special">(</span> <span class="identifier">x</span> <span class="special">)</span> <span class="special">);</span> 250 <span class="special">}</span> 251 252 <span class="keyword">double</span> <span class="identifier">m_gamma</span><span class="special">;</span> 253<span class="special">};</span> 254</pre> 255<p> 256 </p> 257<p> 258 Using the <code class="computeroutput"><span class="identifier">openmp_state</span><span class="special"><</span><span class="identifier">T</span><span class="special">></span></code> state type automatically selects <code class="computeroutput"><span class="identifier">openmp_algebra</span></code> which executes odeint's 259 internal computations on parallel regions. Hence, no manual configuration 260 of the stepper is necessary. At the end of the integration, we use <code class="computeroutput"><span class="identifier">unsplit</span></code> to concatenate the sub-regions 261 back together into a single vector. 262 </p> 263<p> 264</p> 265<pre class="programlisting"><span class="identifier">integrate_n_steps</span><span class="special">(</span> <span class="identifier">runge_kutta4</span><span class="special"><</span><span class="identifier">state_type</span><span class="special">>()</span> <span class="special">,</span> <span class="identifier">phase_chain_omp_state</span><span class="special">(</span> <span class="number">1.2</span> <span class="special">)</span> <span class="special">,</span> 266 <span class="identifier">x_split</span> <span class="special">,</span> <span class="number">0.0</span> <span class="special">,</span> <span class="number">0.01</span> <span class="special">,</span> <span class="number">100</span> <span class="special">);</span> 267<span class="identifier">unsplit</span><span class="special">(</span> <span class="identifier">x_split</span> <span class="special">,</span> <span class="identifier">x</span> <span class="special">);</span> 268</pre> 269<p> 270 </p> 271<div class="note"><table border="0" summary="Note"> 272<tr> 273<td rowspan="2" align="center" valign="top" width="25"><img alt="[Note]" src="../../../../../../../doc/src/images/note.png"></td> 274<th align="left">Note</th> 275</tr> 276<tr><td align="left" valign="top"><p> 277 You don't actually need to use <code class="computeroutput"><span class="identifier">openmp_state</span><span class="special"><</span><span class="identifier">T</span><span class="special">></span></code> for advanced use cases, <code class="computeroutput"><span class="identifier">openmp_algebra</span></code> is simply an alias for 278 <code class="computeroutput"><span class="identifier">openmp_nested_algebra</span><span class="special"><</span><span class="identifier">range_algebra</span><span class="special">></span></code> and supports any model of Random 279 Access Range as the outer, parallel state type, and will use the given 280 algebra on its elements. 281 </p></td></tr> 282</table></div> 283<p> 284 See <a href="https://github.com/headmyshoulder/odeint-v2/blob/master/examples/openmp/phase_chain_omp_state.cpp" target="_top">openmp/phase_chain_omp_state.cpp</a> for 285 the complete example. 286 </p> 287</div> 288<div class="section"> 289<div class="titlepage"><div><div><h4 class="title"> 290<a name="boost_numeric_odeint.tutorial.parallel_computation_with_openmp_and_mpi.mpi"></a><a class="link" href="parallel_computation_with_openmp_and_mpi.html#boost_numeric_odeint.tutorial.parallel_computation_with_openmp_and_mpi.mpi" title="MPI">MPI</a> 291</h4></div></div></div> 292<p> 293 To expand the parallel computation across multiple machines we can use 294 MPI. 295 </p> 296<p> 297 The system function implementation is similar to the OpenMP variant with 298 split data, the main difference being that while OpenMP uses a spawn/join 299 model where everything not explicitly paralleled is only executed in the 300 main thread, in MPI's model each node enters the <code class="computeroutput"><span class="identifier">main</span><span class="special">()</span></code> method independently, diverging based 301 on its rank and synchronizing through message-passing and explicit barriers. 302 </p> 303<p> 304 odeint's MPI support is implemented as an external backend, too. Depending 305 on the MPI implementation the code might need to be compiled with i.e. 306 <code class="literal">mpic++</code>. 307 </p> 308<p> 309</p> 310<pre class="programlisting"><span class="preprocessor">#include</span> <span class="special"><</span><span class="identifier">boost</span><span class="special">/</span><span class="identifier">numeric</span><span class="special">/</span><span class="identifier">odeint</span><span class="special">.</span><span class="identifier">hpp</span><span class="special">></span> 311<span class="preprocessor">#include</span> <span class="special"><</span><span class="identifier">boost</span><span class="special">/</span><span class="identifier">numeric</span><span class="special">/</span><span class="identifier">odeint</span><span class="special">/</span><span class="identifier">external</span><span class="special">/</span><span class="identifier">mpi</span><span class="special">/</span><span class="identifier">mpi</span><span class="special">.</span><span class="identifier">hpp</span><span class="special">></span> 312</pre> 313<p> 314 </p> 315<p> 316 Instead of reading another thread's data, we asynchronously send and receive 317 the relevant data from neighbouring nodes, performing some computation 318 in the interim to hide the latency. 319 </p> 320<p> 321</p> 322<pre class="programlisting"><span class="keyword">struct</span> <span class="identifier">phase_chain_mpi_state</span> 323<span class="special">{</span> 324 <span class="identifier">phase_chain_mpi_state</span><span class="special">(</span> <span class="keyword">double</span> <span class="identifier">gamma</span> <span class="special">=</span> <span class="number">0.5</span> <span class="special">)</span> 325 <span class="special">:</span> <span class="identifier">m_gamma</span><span class="special">(</span> <span class="identifier">gamma</span> <span class="special">)</span> <span class="special">{</span> <span class="special">}</span> 326 327 <span class="keyword">void</span> <span class="keyword">operator</span><span class="special">()(</span> <span class="keyword">const</span> <span class="identifier">state_type</span> <span class="special">&</span><span class="identifier">x</span> <span class="special">,</span> <span class="identifier">state_type</span> <span class="special">&</span><span class="identifier">dxdt</span> <span class="special">,</span> <span class="keyword">double</span> <span class="comment">/* t */</span> <span class="special">)</span> <span class="keyword">const</span> 328 <span class="special">{</span> 329 <span class="keyword">const</span> <span class="identifier">size_t</span> <span class="identifier">M</span> <span class="special">=</span> <span class="identifier">x</span><span class="special">().</span><span class="identifier">size</span><span class="special">();</span> 330 <span class="keyword">const</span> <span class="keyword">bool</span> <span class="identifier">have_left</span> <span class="special">=</span> <span class="identifier">x</span><span class="special">.</span><span class="identifier">world</span><span class="special">.</span><span class="identifier">rank</span><span class="special">()</span> <span class="special">></span> <span class="number">0</span><span class="special">,</span> 331 <span class="identifier">have_right</span> <span class="special">=</span> <span class="identifier">x</span><span class="special">.</span><span class="identifier">world</span><span class="special">.</span><span class="identifier">rank</span><span class="special">()</span> <span class="special"><</span> <span class="identifier">x</span><span class="special">.</span><span class="identifier">world</span><span class="special">.</span><span class="identifier">size</span><span class="special">()-</span><span class="number">1</span><span class="special">;</span> 332 <span class="keyword">double</span> <span class="identifier">x_left</span><span class="special">,</span> <span class="identifier">x_right</span><span class="special">;</span> 333 <span class="identifier">boost</span><span class="special">::</span><span class="identifier">mpi</span><span class="special">::</span><span class="identifier">request</span> <span class="identifier">r_left</span><span class="special">,</span> <span class="identifier">r_right</span><span class="special">;</span> 334 <span class="keyword">if</span><span class="special">(</span> <span class="identifier">have_left</span> <span class="special">)</span> 335 <span class="special">{</span> 336 <span class="identifier">x</span><span class="special">.</span><span class="identifier">world</span><span class="special">.</span><span class="identifier">isend</span><span class="special">(</span> <span class="identifier">x</span><span class="special">.</span><span class="identifier">world</span><span class="special">.</span><span class="identifier">rank</span><span class="special">()-</span><span class="number">1</span> <span class="special">,</span> <span class="number">0</span> <span class="special">,</span> <span class="identifier">x</span><span class="special">().</span><span class="identifier">front</span><span class="special">()</span> <span class="special">);</span> <span class="comment">// send to x_right</span> 337 <span class="identifier">r_left</span> <span class="special">=</span> <span class="identifier">x</span><span class="special">.</span><span class="identifier">world</span><span class="special">.</span><span class="identifier">irecv</span><span class="special">(</span> <span class="identifier">x</span><span class="special">.</span><span class="identifier">world</span><span class="special">.</span><span class="identifier">rank</span><span class="special">()-</span><span class="number">1</span> <span class="special">,</span> <span class="number">0</span> <span class="special">,</span> <span class="identifier">x_left</span> <span class="special">);</span> <span class="comment">// receive from x().back()</span> 338 <span class="special">}</span> 339 <span class="keyword">if</span><span class="special">(</span> <span class="identifier">have_right</span> <span class="special">)</span> 340 <span class="special">{</span> 341 <span class="identifier">x</span><span class="special">.</span><span class="identifier">world</span><span class="special">.</span><span class="identifier">isend</span><span class="special">(</span> <span class="identifier">x</span><span class="special">.</span><span class="identifier">world</span><span class="special">.</span><span class="identifier">rank</span><span class="special">()+</span><span class="number">1</span> <span class="special">,</span> <span class="number">0</span> <span class="special">,</span> <span class="identifier">x</span><span class="special">().</span><span class="identifier">back</span><span class="special">()</span> <span class="special">);</span> <span class="comment">// send to x_left</span> 342 <span class="identifier">r_right</span> <span class="special">=</span> <span class="identifier">x</span><span class="special">.</span><span class="identifier">world</span><span class="special">.</span><span class="identifier">irecv</span><span class="special">(</span> <span class="identifier">x</span><span class="special">.</span><span class="identifier">world</span><span class="special">.</span><span class="identifier">rank</span><span class="special">()+</span><span class="number">1</span> <span class="special">,</span> <span class="number">0</span> <span class="special">,</span> <span class="identifier">x_right</span> <span class="special">);</span> <span class="comment">// receive from x().front()</span> 343 <span class="special">}</span> 344 <span class="keyword">for</span><span class="special">(</span><span class="identifier">size_t</span> <span class="identifier">m</span> <span class="special">=</span> <span class="number">1</span> <span class="special">;</span> <span class="identifier">m</span> <span class="special"><</span> <span class="identifier">M</span><span class="special">-</span><span class="number">1</span> <span class="special">;</span> <span class="special">++</span><span class="identifier">m</span><span class="special">)</span> 345 <span class="special">{</span> 346 <span class="identifier">dxdt</span><span class="special">()[</span><span class="identifier">m</span><span class="special">]</span> <span class="special">=</span> <span class="identifier">coupling_func</span><span class="special">(</span> <span class="identifier">x</span><span class="special">()[</span><span class="identifier">m</span><span class="special">+</span><span class="number">1</span><span class="special">]</span> <span class="special">-</span> <span class="identifier">x</span><span class="special">()[</span><span class="identifier">m</span><span class="special">]</span> <span class="special">)</span> <span class="special">+</span> 347 <span class="identifier">coupling_func</span><span class="special">(</span> <span class="identifier">x</span><span class="special">()[</span><span class="identifier">m</span><span class="special">-</span><span class="number">1</span><span class="special">]</span> <span class="special">-</span> <span class="identifier">x</span><span class="special">()[</span><span class="identifier">m</span><span class="special">]</span> <span class="special">);</span> 348 <span class="special">}</span> 349 <span class="identifier">dxdt</span><span class="special">()[</span><span class="number">0</span><span class="special">]</span> <span class="special">=</span> <span class="identifier">coupling_func</span><span class="special">(</span> <span class="identifier">x</span><span class="special">()[</span><span class="number">1</span><span class="special">]</span> <span class="special">-</span> <span class="identifier">x</span><span class="special">()[</span><span class="number">0</span><span class="special">]</span> <span class="special">);</span> 350 <span class="keyword">if</span><span class="special">(</span> <span class="identifier">have_left</span> <span class="special">)</span> 351 <span class="special">{</span> 352 <span class="identifier">r_left</span><span class="special">.</span><span class="identifier">wait</span><span class="special">();</span> 353 <span class="identifier">dxdt</span><span class="special">()[</span><span class="number">0</span><span class="special">]</span> <span class="special">+=</span> <span class="identifier">coupling_func</span><span class="special">(</span> <span class="identifier">x_left</span> <span class="special">-</span> <span class="identifier">x</span><span class="special">().</span><span class="identifier">front</span><span class="special">()</span> <span class="special">);</span> 354 <span class="special">}</span> 355 <span class="identifier">dxdt</span><span class="special">()[</span><span class="identifier">M</span><span class="special">-</span><span class="number">1</span><span class="special">]</span> <span class="special">=</span> <span class="identifier">coupling_func</span><span class="special">(</span> <span class="identifier">x</span><span class="special">()[</span><span class="identifier">M</span><span class="special">-</span><span class="number">2</span><span class="special">]</span> <span class="special">-</span> <span class="identifier">x</span><span class="special">()[</span><span class="identifier">M</span><span class="special">-</span><span class="number">1</span><span class="special">]</span> <span class="special">);</span> 356 <span class="keyword">if</span><span class="special">(</span> <span class="identifier">have_right</span> <span class="special">)</span> 357 <span class="special">{</span> 358 <span class="identifier">r_right</span><span class="special">.</span><span class="identifier">wait</span><span class="special">();</span> 359 <span class="identifier">dxdt</span><span class="special">()[</span><span class="identifier">M</span><span class="special">-</span><span class="number">1</span><span class="special">]</span> <span class="special">+=</span> <span class="identifier">coupling_func</span><span class="special">(</span> <span class="identifier">x_right</span> <span class="special">-</span> <span class="identifier">x</span><span class="special">().</span><span class="identifier">back</span><span class="special">()</span> <span class="special">);</span> 360 <span class="special">}</span> 361 <span class="special">}</span> 362 363 <span class="keyword">double</span> <span class="identifier">coupling_func</span><span class="special">(</span> <span class="keyword">double</span> <span class="identifier">x</span> <span class="special">)</span> <span class="keyword">const</span> 364 <span class="special">{</span> 365 <span class="keyword">return</span> <span class="identifier">sin</span><span class="special">(</span> <span class="identifier">x</span> <span class="special">)</span> <span class="special">-</span> <span class="identifier">m_gamma</span> <span class="special">*</span> <span class="special">(</span> <span class="number">1.0</span> <span class="special">-</span> <span class="identifier">cos</span><span class="special">(</span> <span class="identifier">x</span> <span class="special">)</span> <span class="special">);</span> 366 <span class="special">}</span> 367 368 <span class="keyword">double</span> <span class="identifier">m_gamma</span><span class="special">;</span> 369<span class="special">};</span> 370</pre> 371<p> 372 </p> 373<p> 374 Analogous to <code class="computeroutput"><span class="identifier">openmp_state</span><span class="special"><</span><span class="identifier">T</span><span class="special">></span></code> we use <code class="computeroutput"><span class="identifier">mpi_state</span><span class="special"><</span> <span class="identifier">InnerState</span><span class="special"><</span><span class="identifier">T</span><span class="special">></span> <span class="special">></span></code>, 375 which automatically selects <code class="computeroutput"><span class="identifier">mpi_nested_algebra</span></code> 376 and the appropriate MPI-oblivious inner algebra (since our inner state 377 is a <code class="computeroutput"><span class="identifier">vector</span></code>, the inner 378 algebra will be <code class="computeroutput"><span class="identifier">range_algebra</span></code> 379 as in the OpenMP example). 380 </p> 381<p> 382</p> 383<pre class="programlisting"><span class="keyword">typedef</span> <span class="identifier">mpi_state</span><span class="special"><</span> <span class="identifier">vector</span><span class="special"><</span><span class="keyword">double</span><span class="special">></span> <span class="special">></span> <span class="identifier">state_type</span><span class="special">;</span> 384</pre> 385<p> 386 </p> 387<p> 388 In the main program we construct a <code class="computeroutput"><span class="identifier">communicator</span></code> 389 which tells us the <code class="computeroutput"><span class="identifier">size</span></code> 390 of the cluster and the current node's <code class="computeroutput"><span class="identifier">rank</span></code> 391 within that. We generate the input data on the master node only, avoiding 392 unnecessary work on the other nodes. Instead of simply copying chunks, 393 <code class="computeroutput"><span class="identifier">split</span></code> acts as a MPI collective 394 function here and sends/receives regions from master to each slave. The 395 input argument is ignored on the slaves, but the master node receives a 396 region in its output and will participate in the computation. 397 </p> 398<p> 399</p> 400<pre class="programlisting"><span class="identifier">boost</span><span class="special">::</span><span class="identifier">mpi</span><span class="special">::</span><span class="identifier">environment</span> <span class="identifier">env</span><span class="special">(</span> <span class="identifier">argc</span> <span class="special">,</span> <span class="identifier">argv</span> <span class="special">);</span> 401<span class="identifier">boost</span><span class="special">::</span><span class="identifier">mpi</span><span class="special">::</span><span class="identifier">communicator</span> <span class="identifier">world</span><span class="special">;</span> 402 403<span class="keyword">const</span> <span class="identifier">size_t</span> <span class="identifier">N</span> <span class="special">=</span> <span class="number">131101</span><span class="special">;</span> 404<span class="identifier">vector</span><span class="special"><</span><span class="keyword">double</span><span class="special">></span> <span class="identifier">x</span><span class="special">;</span> 405<span class="keyword">if</span><span class="special">(</span> <span class="identifier">world</span><span class="special">.</span><span class="identifier">rank</span><span class="special">()</span> <span class="special">==</span> <span class="number">0</span> <span class="special">)</span> 406<span class="special">{</span> 407 <span class="identifier">x</span><span class="special">.</span><span class="identifier">resize</span><span class="special">(</span> <span class="identifier">N</span> <span class="special">);</span> 408 <span class="identifier">boost</span><span class="special">::</span><span class="identifier">random</span><span class="special">::</span><span class="identifier">uniform_real_distribution</span><span class="special"><</span><span class="keyword">double</span><span class="special">></span> <span class="identifier">distribution</span><span class="special">(</span> <span class="number">0.0</span> <span class="special">,</span> <span class="number">2.0</span><span class="special">*</span><span class="identifier">pi</span> <span class="special">);</span> 409 <span class="identifier">boost</span><span class="special">::</span><span class="identifier">random</span><span class="special">::</span><span class="identifier">mt19937</span> <span class="identifier">engine</span><span class="special">(</span> <span class="number">0</span> <span class="special">);</span> 410 <span class="identifier">generate</span><span class="special">(</span> <span class="identifier">x</span><span class="special">.</span><span class="identifier">begin</span><span class="special">()</span> <span class="special">,</span> <span class="identifier">x</span><span class="special">.</span><span class="identifier">end</span><span class="special">()</span> <span class="special">,</span> <span class="identifier">boost</span><span class="special">::</span><span class="identifier">bind</span><span class="special">(</span> <span class="identifier">distribution</span> <span class="special">,</span> <span class="identifier">engine</span> <span class="special">)</span> <span class="special">);</span> 411<span class="special">}</span> 412 413<span class="identifier">state_type</span> <span class="identifier">x_split</span><span class="special">(</span> <span class="identifier">world</span> <span class="special">);</span> 414<span class="identifier">split</span><span class="special">(</span> <span class="identifier">x</span> <span class="special">,</span> <span class="identifier">x_split</span> <span class="special">);</span> 415</pre> 416<p> 417 </p> 418<p> 419 Now that <code class="computeroutput"><span class="identifier">x_split</span></code> contains 420 (only) the local chunk for each node, we start the integration. 421 </p> 422<p> 423 To print the result on the master node, we send the processed data back 424 using <code class="computeroutput"><span class="identifier">unsplit</span></code>. 425 </p> 426<p> 427</p> 428<pre class="programlisting"><span class="identifier">integrate_n_steps</span><span class="special">(</span> <span class="identifier">runge_kutta4</span><span class="special"><</span><span class="identifier">state_type</span><span class="special">>()</span> <span class="special">,</span> <span class="identifier">phase_chain_mpi_state</span><span class="special">(</span> <span class="number">1.2</span> <span class="special">)</span> <span class="special">,</span> 429 <span class="identifier">x_split</span> <span class="special">,</span> <span class="number">0.0</span> <span class="special">,</span> <span class="number">0.01</span> <span class="special">,</span> <span class="number">100</span> <span class="special">);</span> 430<span class="identifier">unsplit</span><span class="special">(</span> <span class="identifier">x_split</span> <span class="special">,</span> <span class="identifier">x</span> <span class="special">);</span> 431</pre> 432<p> 433 </p> 434<div class="note"><table border="0" summary="Note"> 435<tr> 436<td rowspan="2" align="center" valign="top" width="25"><img alt="[Note]" src="../../../../../../../doc/src/images/note.png"></td> 437<th align="left">Note</th> 438</tr> 439<tr><td align="left" valign="top"><p> 440 <code class="computeroutput"><span class="identifier">mpi_nested_algebra</span><span class="special">::</span><span class="identifier">for_each</span></code><em class="replaceable"><code>N</code></em> 441 doesn't use any MPI constructs, it simply calls the inner algebra on 442 the local chunk and the system function is not guarded by any barriers 443 either, so if you don't manually place any (for example in parameter 444 studies cases where the elements are completely independent) you might 445 see the nodes diverging, returning from this call at different times. 446 </p></td></tr> 447</table></div> 448<p> 449 See <a href="https://github.com/headmyshoulder/odeint-v2/blob/master/examples/mpi/phase_chain.cpp" target="_top">mpi/phase_chain.cpp</a> for the complete example. 450 </p> 451</div> 452<div class="section"> 453<div class="titlepage"><div><div><h4 class="title"> 454<a name="boost_numeric_odeint.tutorial.parallel_computation_with_openmp_and_mpi.concepts"></a><a class="link" href="parallel_computation_with_openmp_and_mpi.html#boost_numeric_odeint.tutorial.parallel_computation_with_openmp_and_mpi.concepts" title="Concepts">Concepts</a> 455</h4></div></div></div> 456<div class="toc"><dl class="toc"> 457<dt><span class="section"><a href="parallel_computation_with_openmp_and_mpi.html#boost_numeric_odeint.tutorial.parallel_computation_with_openmp_and_mpi.concepts.mpi_state">MPI 458 State</a></span></dt> 459<dt><span class="section"><a href="parallel_computation_with_openmp_and_mpi.html#boost_numeric_odeint.tutorial.parallel_computation_with_openmp_and_mpi.concepts.openmp_split_state">OpenMP 460 Split State</a></span></dt> 461<dt><span class="section"><a href="parallel_computation_with_openmp_and_mpi.html#boost_numeric_odeint.tutorial.parallel_computation_with_openmp_and_mpi.concepts.splitter">Splitter</a></span></dt> 462</dl></div> 463<div class="section"> 464<div class="titlepage"><div><div><h5 class="title"> 465<a name="boost_numeric_odeint.tutorial.parallel_computation_with_openmp_and_mpi.concepts.mpi_state"></a><a class="link" href="parallel_computation_with_openmp_and_mpi.html#boost_numeric_odeint.tutorial.parallel_computation_with_openmp_and_mpi.concepts.mpi_state" title="MPI State">MPI 466 State</a> 467</h5></div></div></div> 468<p> 469 As used by <code class="computeroutput"><span class="identifier">mpi_nested_algebra</span></code>. 470 </p> 471<h6> 472<a name="boost_numeric_odeint.tutorial.parallel_computation_with_openmp_and_mpi.concepts.mpi_state.h0"></a> 473 <span class="phrase"><a name="boost_numeric_odeint.tutorial.parallel_computation_with_openmp_and_mpi.concepts.mpi_state.notation"></a></span><a class="link" href="parallel_computation_with_openmp_and_mpi.html#boost_numeric_odeint.tutorial.parallel_computation_with_openmp_and_mpi.concepts.mpi_state.notation">Notation</a> 474 </h6> 475<div class="variablelist"> 476<p class="title"><b></b></p> 477<dl class="variablelist"> 478<dt><span class="term"><code class="computeroutput"><span class="identifier">InnerState</span></code></span></dt> 479<dd><p> 480 The inner state type 481 </p></dd> 482<dt><span class="term"><code class="computeroutput"><span class="identifier">State</span></code></span></dt> 483<dd><p> 484 The MPI-state type 485 </p></dd> 486<dt><span class="term"><code class="computeroutput"><span class="identifier">state</span></code></span></dt> 487<dd><p> 488 Object of type <code class="computeroutput"><span class="identifier">State</span></code> 489 </p></dd> 490<dt><span class="term"><code class="computeroutput"><span class="identifier">world</span></code></span></dt> 491<dd><p> 492 Object of type <code class="computeroutput"><span class="identifier">boost</span><span class="special">::</span><span class="identifier">mpi</span><span class="special">::</span><span class="identifier">communicator</span></code> 493 </p></dd> 494</dl> 495</div> 496<h6> 497<a name="boost_numeric_odeint.tutorial.parallel_computation_with_openmp_and_mpi.concepts.mpi_state.h1"></a> 498 <span class="phrase"><a name="boost_numeric_odeint.tutorial.parallel_computation_with_openmp_and_mpi.concepts.mpi_state.valid_expressions"></a></span><a class="link" href="parallel_computation_with_openmp_and_mpi.html#boost_numeric_odeint.tutorial.parallel_computation_with_openmp_and_mpi.concepts.mpi_state.valid_expressions">Valid 499 Expressions</a> 500 </h6> 501<div class="informaltable"><table class="table"> 502<colgroup> 503<col> 504<col> 505<col> 506<col> 507</colgroup> 508<thead><tr> 509<th> 510 <p> 511 Name 512 </p> 513 </th> 514<th> 515 <p> 516 Expression 517 </p> 518 </th> 519<th> 520 <p> 521 Type 522 </p> 523 </th> 524<th> 525 <p> 526 Semantics 527 </p> 528 </th> 529</tr></thead> 530<tbody> 531<tr> 532<td> 533 <p> 534 Construct a state with a communicator 535 </p> 536 </td> 537<td> 538 <p> 539 <code class="computeroutput"><span class="identifier">State</span><span class="special">(</span><span class="identifier">world</span><span class="special">)</span></code> 540 </p> 541 </td> 542<td> 543 <p> 544 <code class="computeroutput"><span class="identifier">State</span></code> 545 </p> 546 </td> 547<td> 548 <p> 549 Constructs the State. 550 </p> 551 </td> 552</tr> 553<tr> 554<td> 555 <p> 556 Construct a state with the default communicator 557 </p> 558 </td> 559<td> 560 <p> 561 <code class="computeroutput"><span class="identifier">State</span><span class="special">()</span></code> 562 </p> 563 </td> 564<td> 565 <p> 566 <code class="computeroutput"><span class="identifier">State</span></code> 567 </p> 568 </td> 569<td> 570 <p> 571 Constructs the State. 572 </p> 573 </td> 574</tr> 575<tr> 576<td> 577 <p> 578 Get the current node's inner state 579 </p> 580 </td> 581<td> 582 <p> 583 <code class="computeroutput"><span class="identifier">state</span><span class="special">()</span></code> 584 </p> 585 </td> 586<td> 587 <p> 588 <code class="computeroutput"><span class="identifier">InnerState</span></code> 589 </p> 590 </td> 591<td> 592 <p> 593 Returns a (const) reference. 594 </p> 595 </td> 596</tr> 597<tr> 598<td> 599 <p> 600 Get the communicator 601 </p> 602 </td> 603<td> 604 <p> 605 <code class="computeroutput"><span class="identifier">state</span><span class="special">.</span><span class="identifier">world</span></code> 606 </p> 607 </td> 608<td> 609 <p> 610 <code class="computeroutput"><span class="identifier">boost</span><span class="special">::</span><span class="identifier">mpi</span><span class="special">::</span><span class="identifier">communicator</span></code> 611 </p> 612 </td> 613<td> 614 <p> 615 See <a href="http://www.boost.org/doc/libs/release/libs/mpi/" target="_top">Boost.MPI</a>. 616 </p> 617 </td> 618</tr> 619</tbody> 620</table></div> 621<h6> 622<a name="boost_numeric_odeint.tutorial.parallel_computation_with_openmp_and_mpi.concepts.mpi_state.h2"></a> 623 <span class="phrase"><a name="boost_numeric_odeint.tutorial.parallel_computation_with_openmp_and_mpi.concepts.mpi_state.models"></a></span><a class="link" href="parallel_computation_with_openmp_and_mpi.html#boost_numeric_odeint.tutorial.parallel_computation_with_openmp_and_mpi.concepts.mpi_state.models">Models</a> 624 </h6> 625<div class="itemizedlist"><ul class="itemizedlist" style="list-style-type: disc; "><li class="listitem"> 626 <code class="computeroutput"><span class="identifier">mpi_state</span><span class="special"><</span><span class="identifier">InnerState</span><span class="special">></span></code> 627 </li></ul></div> 628</div> 629<div class="section"> 630<div class="titlepage"><div><div><h5 class="title"> 631<a name="boost_numeric_odeint.tutorial.parallel_computation_with_openmp_and_mpi.concepts.openmp_split_state"></a><a class="link" href="parallel_computation_with_openmp_and_mpi.html#boost_numeric_odeint.tutorial.parallel_computation_with_openmp_and_mpi.concepts.openmp_split_state" title="OpenMP Split State">OpenMP 632 Split State</a> 633</h5></div></div></div> 634<p> 635 As used by <code class="computeroutput"><span class="identifier">openmp_nested_algebra</span></code>, 636 essentially a Random Access Container with <code class="computeroutput"><span class="identifier">ValueType</span> 637 <span class="special">=</span> <span class="identifier">InnerState</span></code>. 638 </p> 639<h6> 640<a name="boost_numeric_odeint.tutorial.parallel_computation_with_openmp_and_mpi.concepts.openmp_split_state.h0"></a> 641 <span class="phrase"><a name="boost_numeric_odeint.tutorial.parallel_computation_with_openmp_and_mpi.concepts.openmp_split_state.notation"></a></span><a class="link" href="parallel_computation_with_openmp_and_mpi.html#boost_numeric_odeint.tutorial.parallel_computation_with_openmp_and_mpi.concepts.openmp_split_state.notation">Notation</a> 642 </h6> 643<div class="variablelist"> 644<p class="title"><b></b></p> 645<dl class="variablelist"> 646<dt><span class="term"><code class="computeroutput"><span class="identifier">InnerState</span></code></span></dt> 647<dd><p> 648 The inner state type 649 </p></dd> 650<dt><span class="term"><code class="computeroutput"><span class="identifier">State</span></code></span></dt> 651<dd><p> 652 The split state type 653 </p></dd> 654<dt><span class="term"><code class="computeroutput"><span class="identifier">state</span></code></span></dt> 655<dd><p> 656 Object of type <code class="computeroutput"><span class="identifier">State</span></code> 657 </p></dd> 658</dl> 659</div> 660<h6> 661<a name="boost_numeric_odeint.tutorial.parallel_computation_with_openmp_and_mpi.concepts.openmp_split_state.h1"></a> 662 <span class="phrase"><a name="boost_numeric_odeint.tutorial.parallel_computation_with_openmp_and_mpi.concepts.openmp_split_state.valid_expressions"></a></span><a class="link" href="parallel_computation_with_openmp_and_mpi.html#boost_numeric_odeint.tutorial.parallel_computation_with_openmp_and_mpi.concepts.openmp_split_state.valid_expressions">Valid 663 Expressions</a> 664 </h6> 665<div class="informaltable"><table class="table"> 666<colgroup> 667<col> 668<col> 669<col> 670<col> 671</colgroup> 672<thead><tr> 673<th> 674 <p> 675 Name 676 </p> 677 </th> 678<th> 679 <p> 680 Expression 681 </p> 682 </th> 683<th> 684 <p> 685 Type 686 </p> 687 </th> 688<th> 689 <p> 690 Semantics 691 </p> 692 </th> 693</tr></thead> 694<tbody> 695<tr> 696<td> 697 <p> 698 Construct a state for <code class="computeroutput"><span class="identifier">n</span></code> 699 chunks 700 </p> 701 </td> 702<td> 703 <p> 704 <code class="computeroutput"><span class="identifier">State</span><span class="special">(</span><span class="identifier">n</span><span class="special">)</span></code> 705 </p> 706 </td> 707<td> 708 <p> 709 <code class="computeroutput"><span class="identifier">State</span></code> 710 </p> 711 </td> 712<td> 713 <p> 714 Constructs underlying <code class="computeroutput"><span class="identifier">vector</span></code>. 715 </p> 716 </td> 717</tr> 718<tr> 719<td> 720 <p> 721 Get a chunk 722 </p> 723 </td> 724<td> 725 <p> 726 <code class="computeroutput"><span class="identifier">state</span><span class="special">[</span><span class="identifier">i</span><span class="special">]</span></code> 727 </p> 728 </td> 729<td> 730 <p> 731 <code class="computeroutput"><span class="identifier">InnerState</span></code> 732 </p> 733 </td> 734<td> 735 <p> 736 Accesses underlying <code class="computeroutput"><span class="identifier">vector</span></code>. 737 </p> 738 </td> 739</tr> 740<tr> 741<td> 742 <p> 743 Get the number of chunks 744 </p> 745 </td> 746<td> 747 <p> 748 <code class="computeroutput"><span class="identifier">state</span><span class="special">.</span><span class="identifier">size</span><span class="special">()</span></code> 749 </p> 750 </td> 751<td> 752 <p> 753 <code class="computeroutput"><span class="identifier">size_type</span></code> 754 </p> 755 </td> 756<td> 757 <p> 758 Returns size of underlying <code class="computeroutput"><span class="identifier">vector</span></code>. 759 </p> 760 </td> 761</tr> 762</tbody> 763</table></div> 764<h6> 765<a name="boost_numeric_odeint.tutorial.parallel_computation_with_openmp_and_mpi.concepts.openmp_split_state.h2"></a> 766 <span class="phrase"><a name="boost_numeric_odeint.tutorial.parallel_computation_with_openmp_and_mpi.concepts.openmp_split_state.models"></a></span><a class="link" href="parallel_computation_with_openmp_and_mpi.html#boost_numeric_odeint.tutorial.parallel_computation_with_openmp_and_mpi.concepts.openmp_split_state.models">Models</a> 767 </h6> 768<div class="itemizedlist"><ul class="itemizedlist" style="list-style-type: disc; "><li class="listitem"> 769 <code class="computeroutput"><span class="identifier">openmp_state</span><span class="special"><</span><span class="identifier">ValueType</span><span class="special">></span></code> 770 with <code class="computeroutput"><span class="identifier">InnerState</span> <span class="special">=</span> <span class="identifier">vector</span><span class="special"><</span><span class="identifier">ValueType</span><span class="special">></span></code> 771 </li></ul></div> 772</div> 773<div class="section"> 774<div class="titlepage"><div><div><h5 class="title"> 775<a name="boost_numeric_odeint.tutorial.parallel_computation_with_openmp_and_mpi.concepts.splitter"></a><a class="link" href="parallel_computation_with_openmp_and_mpi.html#boost_numeric_odeint.tutorial.parallel_computation_with_openmp_and_mpi.concepts.splitter" title="Splitter">Splitter</a> 776</h5></div></div></div> 777<h6> 778<a name="boost_numeric_odeint.tutorial.parallel_computation_with_openmp_and_mpi.concepts.splitter.h0"></a> 779 <span class="phrase"><a name="boost_numeric_odeint.tutorial.parallel_computation_with_openmp_and_mpi.concepts.splitter.notation"></a></span><a class="link" href="parallel_computation_with_openmp_and_mpi.html#boost_numeric_odeint.tutorial.parallel_computation_with_openmp_and_mpi.concepts.splitter.notation">Notation</a> 780 </h6> 781<div class="variablelist"> 782<p class="title"><b></b></p> 783<dl class="variablelist"> 784<dt><span class="term"><code class="computeroutput"><span class="identifier">Container1</span></code></span></dt> 785<dd><p> 786 The continuous-data container type 787 </p></dd> 788<dt><span class="term"><code class="computeroutput"><span class="identifier">x</span></code></span></dt> 789<dd><p> 790 Object of type <code class="computeroutput"><span class="identifier">Container1</span></code> 791 </p></dd> 792<dt><span class="term"><code class="computeroutput"><span class="identifier">Container2</span></code></span></dt> 793<dd><p> 794 The chunked-data container type 795 </p></dd> 796<dt><span class="term"><code class="computeroutput"><span class="identifier">y</span></code></span></dt> 797<dd><p> 798 Object of type <code class="computeroutput"><span class="identifier">Container2</span></code> 799 </p></dd> 800</dl> 801</div> 802<h6> 803<a name="boost_numeric_odeint.tutorial.parallel_computation_with_openmp_and_mpi.concepts.splitter.h1"></a> 804 <span class="phrase"><a name="boost_numeric_odeint.tutorial.parallel_computation_with_openmp_and_mpi.concepts.splitter.valid_expressions"></a></span><a class="link" href="parallel_computation_with_openmp_and_mpi.html#boost_numeric_odeint.tutorial.parallel_computation_with_openmp_and_mpi.concepts.splitter.valid_expressions">Valid 805 Expressions</a> 806 </h6> 807<div class="informaltable"><table class="table"> 808<colgroup> 809<col> 810<col> 811<col> 812<col> 813</colgroup> 814<thead><tr> 815<th> 816 <p> 817 Name 818 </p> 819 </th> 820<th> 821 <p> 822 Expression 823 </p> 824 </th> 825<th> 826 <p> 827 Type 828 </p> 829 </th> 830<th> 831 <p> 832 Semantics 833 </p> 834 </th> 835</tr></thead> 836<tbody> 837<tr> 838<td> 839 <p> 840 Copy chunks of input to output elements 841 </p> 842 </td> 843<td> 844 <p> 845 <code class="computeroutput"><span class="identifier">split</span><span class="special">(</span><span class="identifier">x</span><span class="special">,</span> 846 <span class="identifier">y</span><span class="special">)</span></code> 847 </p> 848 </td> 849<td> 850 <p> 851 <code class="computeroutput"><span class="keyword">void</span></code> 852 </p> 853 </td> 854<td> 855 <p> 856 Calls <code class="computeroutput"><span class="identifier">split_impl</span><span class="special"><</span><span class="identifier">Container1</span><span class="special">,</span> <span class="identifier">Container2</span><span class="special">>::</span><span class="identifier">split</span><span class="special">(</span><span class="identifier">x</span><span class="special">,</span> <span class="identifier">y</span><span class="special">)</span></code>, splits <code class="computeroutput"><span class="identifier">x</span></code> 857 into <code class="computeroutput"><span class="identifier">y</span><span class="special">.</span><span class="identifier">size</span><span class="special">()</span></code> 858 chunks. 859 </p> 860 </td> 861</tr> 862<tr> 863<td> 864 <p> 865 Join chunks of input elements to output 866 </p> 867 </td> 868<td> 869 <p> 870 <code class="computeroutput"><span class="identifier">unsplit</span><span class="special">(</span><span class="identifier">y</span><span class="special">,</span> 871 <span class="identifier">x</span><span class="special">)</span></code> 872 </p> 873 </td> 874<td> 875 <p> 876 <code class="computeroutput"><span class="keyword">void</span></code> 877 </p> 878 </td> 879<td> 880 <p> 881 Calls <code class="computeroutput"><span class="identifier">unsplit_impl</span><span class="special"><</span><span class="identifier">Container2</span><span class="special">,</span> <span class="identifier">Container1</span><span class="special">>::</span><span class="identifier">unsplit</span><span class="special">(</span><span class="identifier">y</span><span class="special">,</span> <span class="identifier">x</span><span class="special">)</span></code>, assumes <code class="computeroutput"><span class="identifier">x</span></code> 882 is of the correct size <span class="emphasis"><em>σ <code class="computeroutput"><span class="identifier">y</span><span class="special">[</span><span class="identifier">i</span><span class="special">].</span><span class="identifier">size</span><span class="special">()</span></code></em></span>, does not resize 883 <code class="computeroutput"><span class="identifier">x</span></code>. 884 </p> 885 </td> 886</tr> 887</tbody> 888</table></div> 889<h6> 890<a name="boost_numeric_odeint.tutorial.parallel_computation_with_openmp_and_mpi.concepts.splitter.h2"></a> 891 <span class="phrase"><a name="boost_numeric_odeint.tutorial.parallel_computation_with_openmp_and_mpi.concepts.splitter.models"></a></span><a class="link" href="parallel_computation_with_openmp_and_mpi.html#boost_numeric_odeint.tutorial.parallel_computation_with_openmp_and_mpi.concepts.splitter.models">Models</a> 892 </h6> 893<div class="itemizedlist"><ul class="itemizedlist" style="list-style-type: disc; "> 894<li class="listitem"> 895 defined for <code class="computeroutput"><span class="identifier">Container1</span></code> 896 = <a href="http://www.boost.org/doc/libs/release/libs/range/" target="_top">Boost.Range</a> 897 and <code class="computeroutput"><span class="identifier">Container2</span> <span class="special">=</span> 898 <span class="identifier">openmp_state</span></code> 899 </li> 900<li class="listitem"> 901 and <code class="computeroutput"><span class="identifier">Container2</span> <span class="special">=</span> 902 <span class="identifier">mpi_state</span></code>. 903 </li> 904</ul></div> 905<p> 906 To implement splitters for containers incompatible with <a href="http://www.boost.org/doc/libs/release/libs/range/" target="_top">Boost.Range</a>, 907 specialize the <code class="computeroutput"><span class="identifier">split_impl</span></code> 908 and <code class="computeroutput"><span class="identifier">unsplit_impl</span></code> types: 909</p> 910<pre class="programlisting"><span class="keyword">template</span><span class="special"><</span> <span class="keyword">class</span> <span class="identifier">Container1</span><span class="special">,</span> <span class="keyword">class</span> <span class="identifier">Container2</span> <span class="special">,</span> <span class="keyword">class</span> <span class="identifier">Enabler</span> <span class="special">=</span> <span class="keyword">void</span> <span class="special">></span> 911<span class="keyword">struct</span> <span class="identifier">split_impl</span> <span class="special">{</span> 912 <span class="keyword">static</span> <span class="keyword">void</span> <span class="identifier">split</span><span class="special">(</span> <span class="keyword">const</span> <span class="identifier">Container1</span> <span class="special">&</span><span class="identifier">from</span> <span class="special">,</span> <span class="identifier">Container2</span> <span class="special">&</span><span class="identifier">to</span> <span class="special">);</span> 913<span class="special">};</span> 914 915<span class="keyword">template</span><span class="special"><</span> <span class="keyword">class</span> <span class="identifier">Container2</span><span class="special">,</span> <span class="keyword">class</span> <span class="identifier">Container1</span> <span class="special">,</span> <span class="keyword">class</span> <span class="identifier">Enabler</span> <span class="special">=</span> <span class="keyword">void</span> <span class="special">></span> 916<span class="keyword">struct</span> <span class="identifier">unsplit_impl</span> <span class="special">{</span> 917 <span class="keyword">static</span> <span class="keyword">void</span> <span class="identifier">unsplit</span><span class="special">(</span> <span class="keyword">const</span> <span class="identifier">Container2</span> <span class="special">&</span><span class="identifier">from</span> <span class="special">,</span> <span class="identifier">Container1</span> <span class="special">&</span><span class="identifier">to</span> <span class="special">);</span> 918<span class="special">};</span> 919</pre> 920<p> 921 </p> 922</div> 923</div> 924</div> 925<table xmlns:rev="http://www.cs.rpi.edu/~gregod/boost/tools/doc/revision" width="100%"><tr> 926<td align="left"></td> 927<td align="right"><div class="copyright-footer">Copyright © 2009-2015 Karsten Ahnert and Mario Mulansky<p> 928 Distributed under the Boost Software License, Version 1.0. (See accompanying 929 file LICENSE_1_0.txt or copy at <a href="http://www.boost.org/LICENSE_1_0.txt" target="_top">http://www.boost.org/LICENSE_1_0.txt</a>) 930 </p> 931</div></td> 932</tr></table> 933<hr> 934<div class="spirit-nav"> 935<a accesskey="p" href="using_opencl_via_vexcl.html"><img src="../../../../../../../doc/src/images/prev.png" alt="Prev"></a><a accesskey="u" href="../tutorial.html"><img src="../../../../../../../doc/src/images/up.png" alt="Up"></a><a accesskey="h" href="../../index.html"><img src="../../../../../../../doc/src/images/home.png" alt="Home"></a><a accesskey="n" href="all_examples.html"><img src="../../../../../../../doc/src/images/next.png" alt="Next"></a> 936</div> 937</body> 938</html> 939
