J3/03-282
To: J3
From: Bill Long
Date: November 14, 2003
Subject: FCD comment - Error in G format rounding table
--------------------------------------------------------
This paper is based on an email exchange with Neal Gaarder at Cray.
---------------------------------------
Message 1 from Neal:
Bill
I'm having difficulty with the last table in 10.6.4.1 (Gw.d).
IEEE UP and DOWN rounding depends on the sign of the internal
value. In up mode, -1.1 rounds to -1.0, while 1.1 rounds to 2.0.
In down mode, -1.1 rounds to -2.0, while 1.1 rounds to 1.0.
Shouldn't r be sensitive to sign, for up and down?
UP 1 if the internal value is positive
0 if the internal value is negative.
DOWN 0 if the internal value is positive
1 if the internal value is negative.
Neal
----------------------------------
Response to Message 1 by Bill Long:
Hi Neal,
I've sent this to the I/O expert. However, it seems like the change
you suggest makes the ZERO entry wrong.
For UP, the r value for negatives should be the same as for ZERO -> 1
and the r value for positives should be the opposite the one for ZERO -> 1
For DOWN, the r value for positives should be the same as for ZERO -> 0
and the r value for negatives should be the opposite the one for ZERO -> 0.
It looks like if ZERO is correct then the r should be 1 for UP and 0
for DOWN independent of sign.
Do you think the ZERO entry is wrong?
Cheers,
Bill
--------------------------------------------------
Message 2 from Neal:
Bill
I agree that ZERO needs a change. Here is another
way of thought.
>From the 10/2003 draft for UP (r=1)----
d=1 : 0.09 .le. abs(value) .lt. 0.90 ==> F(w-n).1
d=1 : 0.009 .le. abs(value) .lt. 0.09 ==> F(w-n).2
...
d=1 : 0.9 .le. abs(value) .lt. 9. ==> F(w-n).0
When rounding away from zero, as in UP(positive) and
DOWN(negative), 0.9 should not round up to 1 as it can
be output exactly. Hence the .le. should be on the right
side, as follows.
I suggest the following r=1 for UP(positive) and DOWN(negative):
d=1 : 0.09 .lt. abs(value) .le. 0.9 ==> F(w-n).1
d=1 : 0.009 .lt. abs(value) .le. 0.09 ==> F(w-n).2
...
d=1 : 0.9 .lt. abs(value) .le. 9. ==> F(w-n).0
>From the 10/2003 draft for DOWN (r=0)----
d=1 : 0.10 .le. abs(value) .lt. 1.00 ==> F(w-n).1
d=1 : 0.010 .le. abs(value) .lt. 0.10 ==> F(w-n).2
...
d=1 : 1.0 .le. abs(value) .lt. 10. ==> F(w-n).0
I suggest r=0 for truncation: UP(negative), DOWN(positive)
and ZERO(all):
d=1 : 1.000 .le. abs(value) .lt. 10.0 ==> F(w-n).0
d=1 : 0.100 .le. abs(value) .lt. 1.00 ==> F(w-n).1
d=1 : 0.010 .le. abs(value) .lt. 0.10 ==> F(w-n).2
regards,
Neal