Floating point summary (The GNU Awk User’s Guide)

From Get docs
Gawk/docs/latest/Floating-point-summary


16.8 Summary

  • Most computer arithmetic is done using either integers or floating-point values. Standard awk uses double-precision floating-point values.
  • In the early 1990s Barbie mistakenly said, “Math class is tough!” Although math isn’t tough, floating-point arithmetic isn’t the same as pencil-and-paper math, and care must be taken: - Not all numbers can be represented exactly. - Comparing values should use a delta, instead of being done directly with ‘==’ and ‘!=’. - Errors accumulate. - Operations are not always truly associative or distributive.
  • Increasing the accuracy can help, but it is not a panacea.
  • Often, increasing the accuracy and then rounding to the desired number of digits produces reasonable results.
  • Use -M (or --bignum) to enable MPFR arithmetic. Use PREC to set the precision in bits, and ROUNDMODE to set the IEEE 754 rounding mode.
  • With -M, gawk performs arbitrary-precision integer arithmetic using the GMP library. This is faster and more space-efficient than using MPFR for the same calculations.
  • There are several areas with respect to floating-point numbers where gawk disagrees with the POSIX standard. It pays to be aware of them.
  • Overall, there is no need to be unduly suspicious about the results from floating-point arithmetic. The lesson to remember is that floating-point arithmetic is always more complex than arithmetic using pencil and paper. In order to take advantage of the power of floating-point arithmetic, you need to know its limitations and work within them. For most casual use of floating-point arithmetic, you will often get the expected result if you simply round the display of your final results to the correct number of significant decimal digits.
  • As general advice, avoid presenting numerical data in a manner that implies better precision than is actually the case.