1<html> 2<head> 3<meta http-equiv="Content-Type" content="text/html; charset=UTF-8"> 4<title>Numerical Differentiation</title> 5<link rel="stylesheet" href="../math.css" type="text/css"> 6<meta name="generator" content="DocBook XSL Stylesheets V1.79.1"> 7<link rel="home" href="../index.html" title="Math Toolkit 2.12.0"> 8<link rel="up" href="../quadrature.html" title="Chapter 13. Quadrature and Differentiation"> 9<link rel="prev" href="wavelet_transforms.html" title="Wavelet Transforms"> 10<link rel="next" href="autodiff.html" title="Automatic Differentiation"> 11</head> 12<body bgcolor="white" text="black" link="#0000FF" vlink="#840084" alink="#0000FF"> 13<table cellpadding="2" width="100%"><tr> 14<td valign="top"><img alt="Boost C++ Libraries" width="277" height="86" src="../../../../../boost.png"></td> 15<td align="center"><a href="../../../../../index.html">Home</a></td> 16<td align="center"><a href="../../../../../libs/libraries.htm">Libraries</a></td> 17<td align="center"><a href="http://www.boost.org/users/people.html">People</a></td> 18<td align="center"><a href="http://www.boost.org/users/faq.html">FAQ</a></td> 19<td align="center"><a href="../../../../../more/index.htm">More</a></td> 20</tr></table> 21<hr> 22<div class="spirit-nav"> 23<a accesskey="p" href="wavelet_transforms.html"><img src="../../../../../doc/src/images/prev.png" alt="Prev"></a><a accesskey="u" href="../quadrature.html"><img src="../../../../../doc/src/images/up.png" alt="Up"></a><a accesskey="h" href="../index.html"><img src="../../../../../doc/src/images/home.png" alt="Home"></a><a accesskey="n" href="autodiff.html"><img src="../../../../../doc/src/images/next.png" alt="Next"></a> 24</div> 25<div class="section"> 26<div class="titlepage"><div><div><h2 class="title" style="clear: both"> 27<a name="math_toolkit.diff"></a><a class="link" href="diff.html" title="Numerical Differentiation">Numerical Differentiation</a> 28</h2></div></div></div> 29<h4> 30<a name="math_toolkit.diff.h0"></a> 31 <span class="phrase"><a name="math_toolkit.diff.synopsis"></a></span><a class="link" href="diff.html#math_toolkit.diff.synopsis">Synopsis</a> 32 </h4> 33<pre class="programlisting"><span class="preprocessor">#include</span> <span class="special"><</span><span class="identifier">boost</span><span class="special">/</span><span class="identifier">math</span><span class="special">/</span><span class="identifier">differentiation</span><span class="special">/</span><span class="identifier">finite_difference</span><span class="special">.</span><span class="identifier">hpp</span><span class="special">></span> 34 35 36<span class="keyword">namespace</span> <span class="identifier">boost</span> <span class="special">{</span> <span class="keyword">namespace</span> <span class="identifier">math</span> <span class="special">{</span> <span class="keyword">namespace</span> <span class="identifier">differentiation</span> <span class="special">{</span> 37 38 <span class="keyword">template</span> <span class="special"><</span><span class="keyword">class</span> <span class="identifier">F</span><span class="special">,</span> <span class="keyword">class</span> <span class="identifier">Real</span><span class="special">></span> 39 <span class="identifier">Real</span> <span class="identifier">complex_step_derivative</span><span class="special">(</span><span class="keyword">const</span> <span class="identifier">F</span> <span class="identifier">f</span><span class="special">,</span> <span class="identifier">Real</span> <span class="identifier">x</span><span class="special">);</span> 40 41 <span class="keyword">template</span> <span class="special"><</span><span class="keyword">class</span> <span class="identifier">F</span><span class="special">,</span> <span class="keyword">class</span> <span class="identifier">Real</span><span class="special">,</span> <span class="identifier">size_t</span> <span class="identifier">order</span> <span class="special">=</span> <span class="number">6</span><span class="special">></span> 42 <span class="identifier">Real</span> <span class="identifier">finite_difference_derivative</span><span class="special">(</span><span class="keyword">const</span> <span class="identifier">F</span> <span class="identifier">f</span><span class="special">,</span> <span class="identifier">Real</span> <span class="identifier">x</span><span class="special">,</span> <span class="identifier">Real</span><span class="special">*</span> <span class="identifier">error</span> <span class="special">=</span> <span class="keyword">nullptr</span><span class="special">);</span> 43 44<span class="special">}}}</span> <span class="comment">// namespaces</span> 45</pre> 46<h4> 47<a name="math_toolkit.diff.h1"></a> 48 <span class="phrase"><a name="math_toolkit.diff.description"></a></span><a class="link" href="diff.html#math_toolkit.diff.description">Description</a> 49 </h4> 50<p> 51 The function <code class="computeroutput"><span class="identifier">finite_difference_derivative</span></code> 52 calculates a finite-difference approximation to the derivative of a function 53 <span class="emphasis"><em>f</em></span> at point <span class="emphasis"><em>x</em></span>. A basic usage is 54 </p> 55<pre class="programlisting"><span class="keyword">auto</span> <span class="identifier">f</span> <span class="special">=</span> <span class="special">[](</span><span class="keyword">double</span> <span class="identifier">x</span><span class="special">)</span> <span class="special">{</span> <span class="keyword">return</span> <span class="identifier">std</span><span class="special">::</span><span class="identifier">exp</span><span class="special">(</span><span class="identifier">x</span><span class="special">);</span> <span class="special">};</span> 56<span class="keyword">double</span> <span class="identifier">x</span> <span class="special">=</span> <span class="number">1.7</span><span class="special">;</span> 57<span class="keyword">double</span> <span class="identifier">dfdx</span> <span class="special">=</span> <span class="identifier">finite_difference_derivative</span><span class="special">(</span><span class="identifier">f</span><span class="special">,</span> <span class="identifier">x</span><span class="special">);</span> 58</pre> 59<p> 60 Finite differencing is complicated, as differentiation is <span class="emphasis"><em>infinitely</em></span> 61 ill-conditioned. In addition, for any function implemented in finite-precision 62 arithmetic, the "true" derivative is <span class="emphasis"><em>zero</em></span> almost 63 everywhere, and undefined at representables. However, some tricks allow for 64 reasonable results to be obtained in many cases. 65 </p> 66<p> 67 There are two sources of error from finite differences: One, the truncation 68 error arising from using a finite number of samples to cancel out higher order 69 terms in the Taylor series. The second is the roundoff error involved in evaluating 70 the function. The truncation error goes to zero as <span class="emphasis"><em>h</em></span> → 71 0, but the roundoff error becomes unbounded. By balancing these two sources 72 of error, we can choose a value of <span class="emphasis"><em>h</em></span> that minimizes the 73 maximum total error. For this reason boost's <code class="computeroutput"><span class="identifier">finite_difference_derivative</span></code> 74 does not require the user to input a stepsize. For more details about the theoretical 75 error analysis involved in finite-difference approximations to the derivative, 76 see <a href="http://web.archive.org/web/20150420195907/http://www.uio.no/studier/emner/matnat/math/MAT-INF1100/h08/kompendiet/diffint.pdf" target="_top">here</a>. 77 </p> 78<p> 79 Despite the effort that has went into choosing a reasonable value of <span class="emphasis"><em>h</em></span>, 80 the problem is still fundamentally ill-conditioned, and hence an error estimate 81 is essential. It can be queried as follows 82 </p> 83<pre class="programlisting"><span class="keyword">double</span> <span class="identifier">error_estimate</span><span class="special">;</span> 84<span class="keyword">double</span> <span class="identifier">d</span> <span class="special">=</span> <span class="identifier">finite_difference_derivative</span><span class="special">(</span><span class="identifier">f</span><span class="special">,</span> <span class="identifier">x</span><span class="special">,</span> <span class="special">&</span><span class="identifier">error_estimate</span><span class="special">);</span> 85</pre> 86<p> 87 N.B.: Producing an error estimate requires additional function evaluations 88 and as such is slower than simple evaluation of the derivative. It also expands 89 the domain over which the function must be differentiable and requires the 90 function to have two more continuous derivatives. The error estimate is computed 91 under the assumption that <span class="emphasis"><em>f</em></span> is evaluated to 1ULP. This 92 might seem an extreme assumption, but it is the only sensible one, as the routine 93 cannot know the functions rounding error. If the function cannot be evaluated 94 with very great accuracy, Lanczos's smoothing differentiation is recommended 95 as an alternative. 96 </p> 97<p> 98 The default order of accuracy is 6, which reflects that fact that people tend 99 to be interested in functions with many continuous derivatives. If your function 100 does not have 7 continuous derivatives, is may be of interest to use a lower 101 order method, which can be achieved via (say) 102 </p> 103<pre class="programlisting"><span class="keyword">double</span> <span class="identifier">d</span> <span class="special">=</span> <span class="identifier">finite_difference_derivative</span><span class="special"><</span><span class="keyword">decltype</span><span class="special">(</span><span class="identifier">f</span><span class="special">),</span> <span class="identifier">Real</span><span class="special">,</span> <span class="number">2</span><span class="special">>(</span><span class="identifier">f</span><span class="special">,</span> <span class="identifier">x</span><span class="special">);</span> 104</pre> 105<p> 106 This requests a second-order accurate derivative be computed. 107 </p> 108<p> 109 It is emphatically <span class="emphasis"><em>not</em></span> the case that higher order methods 110 always give higher accuracy for smooth functions. Higher order methods require 111 more addition of positive and negative terms, which can lead to catastrophic 112 cancellation. A function which is very good at making a mockery of finite-difference 113 differentiation is exp(x)/(cos(x)<sup>3</sup> + sin(x)<sup>3</sup>). Differentiating this function 114 by <code class="computeroutput"><span class="identifier">finite_difference_derivative</span></code> 115 in double precision at <span class="emphasis"><em>x=5.5</em></span> gives zero correct digits 116 at order 4, 6, and 8, but recovers 5 correct digits at order 2. These are dangerous 117 waters; use the error estimates to tread carefully. 118 </p> 119<p> 120 For a finite-difference method of order <span class="emphasis"><em>k</em></span>, the error is 121 <span class="emphasis"><em>C</em></span> ε<sup>k/k+1</sup>. In the limit <span class="emphasis"><em>k</em></span> → 122 ∞, we see that the error tends to ε, recovering the full precision 123 for the type. However, this ignores the fact that higher-order methods require 124 subtracting more nearly-equal (perhaps noisy) terms, so the constant <span class="emphasis"><em>C</em></span> 125 grows with <span class="emphasis"><em>k</em></span>. Since <span class="emphasis"><em>C</em></span> grows quickly 126 and ε<sup>k/k+1</sup> approaches ε slowly, we can see there is a compromise 127 between high-order accuracy and conditioning of the difference quotient. In 128 practice we have found that <span class="emphasis"><em>k=6</em></span> seems to be a good compromise 129 between the two (and have made this the default), but users are encouraged 130 to examine the error estimates to choose an optimal order of accuracy for the 131 given problem. 132 </p> 133<div class="table"> 134<a name="math_toolkit.diff.id"></a><p class="title"><b>Table 13.1. Cost of Finite-Difference Numerical Differentiation</b></p> 135<div class="table-contents"><table class="table" summary="Cost of Finite-Difference Numerical Differentiation"> 136<colgroup> 137<col> 138<col> 139<col> 140<col> 141<col> 142</colgroup> 143<thead><tr> 144<th> 145 <p> 146 Order of Accuracy 147 </p> 148 </th> 149<th> 150 <p> 151 Function Evaluations 152 </p> 153 </th> 154<th> 155 <p> 156 Error 157 </p> 158 </th> 159<th> 160 <p> 161 Continuous Derivatives Required for Error Estimate to Hold 162 </p> 163 </th> 164<th> 165 <p> 166 Additional Function Evaluations to Produce Error Estimates 167 </p> 168 </th> 169</tr></thead> 170<tbody> 171<tr> 172<td> 173 <p> 174 1 175 </p> 176 </td> 177<td> 178 <p> 179 2 180 </p> 181 </td> 182<td> 183 <p> 184 ε<sup>1/2</sup> 185 </p> 186 </td> 187<td> 188 <p> 189 2 190 </p> 191 </td> 192<td> 193 <p> 194 1 195 </p> 196 </td> 197</tr> 198<tr> 199<td> 200 <p> 201 2 202 </p> 203 </td> 204<td> 205 <p> 206 2 207 </p> 208 </td> 209<td> 210 <p> 211 ε<sup>2/3</sup> 212 </p> 213 </td> 214<td> 215 <p> 216 3 217 </p> 218 </td> 219<td> 220 <p> 221 2 222 </p> 223 </td> 224</tr> 225<tr> 226<td> 227 <p> 228 4 229 </p> 230 </td> 231<td> 232 <p> 233 4 234 </p> 235 </td> 236<td> 237 <p> 238 ε<sup>4/5</sup> 239 </p> 240 </td> 241<td> 242 <p> 243 5 244 </p> 245 </td> 246<td> 247 <p> 248 2 249 </p> 250 </td> 251</tr> 252<tr> 253<td> 254 <p> 255 6 256 </p> 257 </td> 258<td> 259 <p> 260 6 261 </p> 262 </td> 263<td> 264 <p> 265 ε<sup>6/7</sup> 266 </p> 267 </td> 268<td> 269 <p> 270 7 271 </p> 272 </td> 273<td> 274 <p> 275 2 276 </p> 277 </td> 278</tr> 279<tr> 280<td> 281 <p> 282 8 283 </p> 284 </td> 285<td> 286 <p> 287 8 288 </p> 289 </td> 290<td> 291 <p> 292 ε<sup>8/9</sup> 293 </p> 294 </td> 295<td> 296 <p> 297 9 298 </p> 299 </td> 300<td> 301 <p> 302 2 303 </p> 304 </td> 305</tr> 306</tbody> 307</table></div> 308</div> 309<br class="table-break"><p> 310 Given all the caveats which must be kept in mind for successful use of finite-difference 311 differentiation, it is reasonable to try to avoid it if possible. Boost provides 312 two possibilities: The Chebyshev transform (see <a class="link" href="sf_poly/chebyshev.html" title="Chebyshev Polynomials">here</a>) 313 and the complex step derivative. If your function is the restriction to the 314 real line of a holomorphic function which takes real values at real argument, 315 then the <span class="bold"><strong>complex step derivative</strong></span> can be used. 316 The idea is very simple: Since <span class="emphasis"><em>f</em></span> is complex-differentiable, 317 <span class="emphasis"><em>f(x+ⅈ h) = f(x) + ⅈ hf'(x) - h<sup>2</sup>f''(x) + ��(h<sup>3</sup>)</em></span>. 318 As long as <span class="emphasis"><em>f(x)</em></span> ∈ ℝ, then <span class="emphasis"><em>f'(x) 319 = ℑ f(x+ⅈ h)/h + ��(h<sup>2</sup>)</em></span>. This method requires a single 320 complex function evaluation and is not subject to the catastrophic subtractive 321 cancellation that plagues finite-difference calculations. 322 </p> 323<p> 324 An example usage: 325 </p> 326<pre class="programlisting"><span class="keyword">double</span> <span class="identifier">x</span> <span class="special">=</span> <span class="number">7.2</span><span class="special">;</span> 327<span class="keyword">double</span> <span class="identifier">e_prime</span> <span class="special">=</span> <span class="identifier">complex_step_derivative</span><span class="special">(</span><span class="identifier">std</span><span class="special">::</span><span class="identifier">exp</span><span class="special"><</span><span class="identifier">std</span><span class="special">::</span><span class="identifier">complex</span><span class="special"><</span><span class="keyword">double</span><span class="special">>>,</span> <span class="identifier">x</span><span class="special">);</span> 328</pre> 329<p> 330 References: 331 </p> 332<div class="itemizedlist"><ul class="itemizedlist" style="list-style-type: disc; "> 333<li class="listitem"> 334 Squire, William, and George Trapp. <span class="emphasis"><em>Using complex variables to 335 estimate derivatives of real functions.</em></span> Siam Review 40.1 (1998): 336 110-112. 337 </li> 338<li class="listitem"> 339 Fornberg, Bengt. <span class="emphasis"><em>Generation of finite difference formulas on 340 arbitrarily spaced grids.</em></span> Mathematics of computation 51.184 341 (1988): 699-706. 342 </li> 343<li class="listitem"> 344 Corless, Robert M., and Nicolas Fillion. <span class="emphasis"><em>A graduate introduction 345 to numerical methods.</em></span> AMC 10 (2013): 12. 346 </li> 347</ul></div> 348</div> 349<table xmlns:rev="http://www.cs.rpi.edu/~gregod/boost/tools/doc/revision" width="100%"><tr> 350<td align="left"></td> 351<td align="right"><div class="copyright-footer">Copyright © 2006-2019 Nikhar 352 Agrawal, Anton Bikineev, Paul A. Bristow, Marco Guazzone, Christopher Kormanyos, 353 Hubert Holin, Bruno Lalande, John Maddock, Jeremy Murphy, Matthew Pulver, Johan 354 Råde, Gautam Sewani, Benjamin Sobotta, Nicholas Thompson, Thijs van den Berg, 355 Daryle Walker and Xiaogang Zhang<p> 356 Distributed under the Boost Software License, Version 1.0. (See accompanying 357 file LICENSE_1_0.txt or copy at <a href="http://www.boost.org/LICENSE_1_0.txt" target="_top">http://www.boost.org/LICENSE_1_0.txt</a>) 358 </p> 359</div></td> 360</tr></table> 361<hr> 362<div class="spirit-nav"> 363<a accesskey="p" href="wavelet_transforms.html"><img src="../../../../../doc/src/images/prev.png" alt="Prev"></a><a accesskey="u" href="../quadrature.html"><img src="../../../../../doc/src/images/up.png" alt="Up"></a><a accesskey="h" href="../index.html"><img src="../../../../../doc/src/images/home.png" alt="Home"></a><a accesskey="n" href="autodiff.html"><img src="../../../../../doc/src/images/next.png" alt="Next"></a> 364</div> 365</body> 366</html> 367
