Re: Float Pt Precision

• From: Phil Carmody <carmody cpd ntc nokia com>
• To: axp-list redhat com
• Subject: Re: Float Pt Precision
• Date: Fri, 30 Jul 1999 12:18:37 +0100 (WET DST)

```On Thu, 29 Jul 1999, EXT Uncle George wrote:

> Been trying to determine why the double's between an Intel Box & and
> Alpha box are different on postgresql.
>
> The algoritms:
> 1a)    t1.q = result.A * pt1.x;
> 1b)   t2.q = pt1.y - t1.q;
>
> and
>
> 2)    t3.q = result.C = pt1.y - result.A * pt1.x;
>
> will produce the same results on the Alpha    0x402ec4ec4ec4ec50,
> but will produce different results in the intel    0x402ec4ec4ec4ec50,
> 0x402ec4ec4ec4ec54

OK, so this is calculating the y axis intercept between two known points
using one point and the gradiant.

What do the alpha and x86 grok from the following when the bracketted
expressions are (a) precalulated and stored in temporaries (b) as is.

// optionally
double temp = 1.0 / (pt2.x - pt1.x);
// or temp = (pt2.x - pt1.x)
result.A = (pt2.y - pt1.y) * temp;
// or                      / temp
result.C = (pt1.y * pt2.x - pt2.y * pt1.x) * temp;
// or                                      / temp

which basically takes the weighted mean of the two y values.

Here on my ultrasparc, I get the same answer for each of my expressions,

result->C=15.3846153846154(402ec4ec4ec4ec50)    (intel v1 and alpha both)
result->C=15.3846153846154(402ec4ec4ec4ec54)    (intel v2)
12 3456789012345 12345678901234__
should be 4e
The decimal expression is correct to the precision shown.

so your expression has 2 or 6 ulp error (6 for intel)

My sparc gives get :
my expression:
one expression	15.384615384615385025313116784673(402ec4ec4ec4ec4f)
via temporary 	15.384615384615385025313116784673(402ec4ec4ec4ec4f)
12 34567890123456^^^              123456789012345_
should be 461                   should be e
1 ulp

one expression	15.384615384615358379960525780916(402ec4ec4ec4ec40)
via temporary	15.384615384615358379960525780916(402ec4ec4ec4ec40)
12 3456789012345^^^^              123456789012345_
should be    8461                   should be e
14 ulp

Well, I'm not sure what this tells anyone, but it looks like the sparc
threw its guard digits out the window!
My expression seems more stable, but I'll defer to constructive criticism
from the direction of NIST, for example! (he knows what he's talking baout
guys!)

As with anything, if you want the greater accuracy and/or precision,
there's a cost. Mine takes more multiplies, is it really time critical?

If there's a conclusion, it appears that the alpha is the winner!

Cheerio,
Phil

--
Phil Carmody.
Not speaking for or on behalf of Nokia Telecommunications

How many things served us but yesterday as articles of faith, which today
we deem but fables?
-- Michel Eyquem de Montaigne (1533-1592) Essays, bk. 1, ch. 26,
"It Is Folly to Refer Truth or Falsehood to Our Sufficiency"

```