Because Format is often used as a means of rounding numbers to some desired precision
and because that rounding often creates questions
about the accuracy of the rounding
this seems to be a good place to discuss how rounding works,
and why it is that we sometimes see some unexpected results.
There is nothing within A+ that imposes any special rules on the rounding of numbers.
Rounding is done entirely using IEEE symmetric rounding rules.
That is, if a number lies halfway between two other numbers to which it can legitimately be
equally rounded either way,
it is rounded to the one ending in an even digit.
Therefore, a value of 22.5, for instance, could logically be rounded to either 22 or 23;
the IEEE rules tell us that it should be rounded to 22, because that is the number which
ends with an even digit.
As further examples,
if we are starting with values which are exactly halfway between two integers,
and rounding them to integers:
- 0.5 should round down to an integer value of 0
(not to 1, because 1 is an odd number);
- 1.5 should round up to an even integer value of 2;
- 2.5 should round down to an even integer value of 2;
- 3.5 should round up to an even integer value of 4; and
- 4.5 should round down to an even integer value of 4.
In A+ terms, this would be shown as follows:
4î.5 1.5 2.5 3.5 4.5 ã This example shows IEEE rounding.
0 2 2 4 4
In actual practice, however, machine approximation of decimal numbers and machine rounding may obscure the regular IEEE rules.
For example, given the following vector:
vû1.055 1.155 1.255 1.355 1.455 1.555 1.655 1.755 1.855 1.955
IEEE rules dictate that rounding it to two decimal positions should cause the
"n.n55" values to round to "n.n6"
(and if we had "n.n45" values, they should round to "n.n4").
However, that's not necessarily what happens.
Let's explore what happens, and why.
Original
Value | IEEE
Rounding | 6.2îv
Rounding | Actual Internal
Value |
1.055 | 1.06 | 1.05 | 1.05499999999999994... |
1.155 | 1.16 | 1.16 | 1.15500000000000003... |
1.255 | 1.26 | 1.25 | 1.25499999999999989... |
1.355 | 1.36 | 1.35 | 1.35499999999999998... |
1.455 | 1.46 | 1.46 | 1.45500000000000007... |
1.555 | 1.56 | 1.55 | 1.55499999999999994... |
1.655 | 1.66 | 1.66 | 1.65500000000000003... |
1.755 | 1.76 | 1.75 | 1.75499999999999989... |
1.855 | 1.86 | 1.85 | 1.85499999999999998... |
1.955 | 1.96 | 1.96 | 1.95500000000000007... |
Notice that using 6.2îv will round
the values in v to two decimal places...
but it doesn't necessarily give us what we expect from the IEEE rounding rules.
Some of the results look okay, but realize that the internal values are imprecise for
all of our values,
and some of them just happen to round the way that we want them to.)
The reason for this is that we are doing decimal operations on a hexadecimal machine,
so there will always be some errors in the internal representation of many of the numbers
that we deal with every day. It is simply not possible to represent these values
exactly on a digital machine.
In the same way that we simply cannot represent exactly one-third
as a decimal number, we also cannot represent many other decimal numbers in hexadecimal
and be able to convert them back to exactly the number that we started with.
Therefore, a value of "1.055" actually gets represented within the machine
as "1.05499999999999994...", so that value is correctly rounded down to "1.05",
while a value of "1.155" is actually seen by the machine as "1.15500000000000003...",
so that value is correctly rounded up to "1.16".
Although the first value is rounded to "1.05" instead of the desired IEEE rounding value
of "1.06", realize that the rounding is correct based upon the internal value
that is seen by the hardware.
These internal values are incorrect, but they are as close an approximation of
the exact values as we can get on this hardware.
So why don't we see this imprecision at every step of our calculations? ...Simply because
the printing precision normally masks this error, and over the course of normal work,
these errors tend to cancel out.
Before you get too concerned about this imprecision, realize that the
Comparison Tolerance in A+
(the amount by which two values may differ and still be considered to be equal)
is set to 1e-13 (or 0.0000000000001);
that is a tolerance of one part
(of error) in ten trillion. To put this into perspective, if our measurements
were to represent distance, a measurement of 250,000 miles (approximately the
distance from the earth to the moon) could be carried out within an error
of no more than one-third of the thickness of a piece of copier paper.
For most operations, this is deemed to be "close enough."
Also realize that rounding of values through the use of the Format function
is typically done just for display of final output
(where 16 digits of precision would be inappropriate anyway).
Calculations prior to that final rounding step are done at full precision.
Finally, we just want to emphasize again that none of the rounding that you see
through A+ is any different than it would be in other environments.
A point that may make it seem different is the ease with which you can
adjust the precision and look at alternate views of the same values.
This is a general hardware numeric conversion issue, not an A+ issue.