Gawk/Inexact-representation

Next: Comparing FP Values, Up: Inexactness of computations   [Contents][Index]

16.4.1.1 Many Numbers Cannot Be Represented Exactly

So, before you start to write any code, you should think about what you really want and what’s really happening. Consider the two numbers in the following example:

x = 0.875             # 1/2 + 1/4 + 1/8
y = 0.425

Unlike the number in y, the number stored in x is exactly representable in binary because it can be written as a finite sum of one or more fractions whose denominators are all powers of two. When gawk reads a floating-point number from program source, it automatically rounds that number to whatever precision your machine supports. If you try to print the numeric content of a variable using an output format string of "%.17g", it may not produce the same number as you assigned to it:

$ gawk 'BEGIN { x = 0.875; y = 0.425
>               printf("%0.17g, %0.17g\n", x, y) }'
-| 0.875, 0.42499999999999999

Often the error is so small you do not even notice it, and if you do, you can always specify how much precision you would like in your output. Usually this is a format string like "%.15g", which, when used in the previous example, produces an output identical to the input.