J3/03-282

To: J3
From: Bill Long
Date: November 14, 2003
Subject: FCD comment - Error in G format rounding table
--------------------------------------------------------


This paper is based on an email exchange with Neal Gaarder at Cray.

---------------------------------------
Message 1 from Neal:

Bill

I'm having difficulty with the last table in 10.6.4.1 (Gw.d).


IEEE UP and DOWN rounding depends on the sign of the internal
value.  In up mode, -1.1 rounds to -1.0, while 1.1 rounds to 2.0.
In down mode, -1.1 rounds to -2.0, while 1.1 rounds to 1.0.
Shouldn't r be sensitive to sign, for up and down?

   UP   1 if the internal value is positive
        0 if the internal value is negative.

   DOWN 0 if the internal value is positive
        1 if the internal value is negative.


Neal
----------------------------------
Response to Message 1 by Bill Long:

Hi Neal,

I've sent this to the I/O expert.  However, it seems like the change
you suggest makes the ZERO entry wrong.

For UP, the r value for negatives should be the same as for ZERO -> 1
    and the r value for positives should be the opposite the one for ZERO -> 1

For DOWN, the r value for positives should be the same as for ZERO -> 0
     and  the r value for negatives should be the opposite the one for ZERO -> 0.

It looks like if ZERO is correct then the r should be 1 for UP and 0
for DOWN independent of sign.

Do you think the ZERO entry is wrong?

Cheers,
Bill

--------------------------------------------------

Message 2 from Neal:

Bill

I agree that ZERO needs a change.  Here is another
way of thought.

>From the 10/2003 draft for UP (r=1)----
    d=1 : 0.09  .le. abs(value) .lt. 0.90  ==> F(w-n).1
    d=1 : 0.009 .le. abs(value) .lt. 0.09  ==> F(w-n).2
    ...
    d=1 : 0.9   .le. abs(value) .lt. 9.    ==> F(w-n).0

When rounding away from zero, as in UP(positive) and
DOWN(negative), 0.9 should not round up to 1 as it can
be output exactly.  Hence the .le. should be on the right
side, as follows.

I suggest the following r=1 for UP(positive) and DOWN(negative):

    d=1 : 0.09  .lt. abs(value) .le. 0.9   ==> F(w-n).1
    d=1 : 0.009 .lt. abs(value) .le. 0.09  ==> F(w-n).2
    ...
    d=1 : 0.9   .lt. abs(value) .le. 9.    ==> F(w-n).0


>From the 10/2003 draft for DOWN (r=0)----
    d=1 : 0.10  .le. abs(value) .lt. 1.00  ==> F(w-n).1
    d=1 : 0.010 .le. abs(value) .lt. 0.10  ==> F(w-n).2
    ...
    d=1 : 1.0   .le. abs(value) .lt. 10.   ==> F(w-n).0

I suggest r=0 for truncation: UP(negative), DOWN(positive)
and ZERO(all):
    d=1 : 1.000 .le. abs(value) .lt. 10.0  ==> F(w-n).0
    d=1 : 0.100 .le. abs(value) .lt. 1.00  ==> F(w-n).1
    d=1 : 0.010 .le. abs(value) .lt. 0.10  ==> F(w-n).2


regards,
Neal