Monday, July 01, 2013

Rounding error in Java when converting strings to doubles

I came across a rounding error when I was running an XQuery.

Here is the error:

xquery version "1.0";
number(3.1) + number(3.2)

which returned:

        6.300000000000001

Not the expected value of 6.3.  Note that the XQuery function number() returns double precision data.

After a note from Eric Bruchez he suggested I cast the numbers to decimal:

xquery version "1.0";
xs:decimal(3.1) + xs:decimal(3.2)

and the problem seems to go away.  He also showed that he could reproduce this error in other Java JVM languages like Scala.

Let me know if anyone else has seen this problem before and has any other suggestions for a fix.

Thanks! - Dan

No comments: