Lines Matching refs:precision

113 (a) the operands have a higher precision than the current setting of the
114     precision control (PC) flags.
119     operands were rounded to the current precision before the arithmetic
242 precision of the argument x; e.g. an argument of pi/2-(1e-10) which is
265 later) for 'float' variables (24 bit precision numbers) when precision
267 bit precision numbers) when precision control is set to 53 bits (a
269 variables when precision control is set to 64 bits).
273 for pi which is accurate to more than 128 bits precision. As a
289 give much degraded precision. For example, the integer number
292 example) of this number to 64 bits precision it would be necessary to
293 have a value of pi which had about 150 bits precision. The FPU
294 emulator computes the result to about 42.6 bits precision (the correct