Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Possible floating point error?
#4
(05-30-2024, 01:02 AM)TerryRitchie Wrote: For example. The result of x should be 1 but instead the value .9999999 is given:

f = 2 / 454
x = 227 * f
That's not a bug, that's just how floating point works. The number is stored in scientific notation with about 7 to 8 digits of precision (23 binary digits). If you take a look at the value you calculated in base 2, it's a repeating number of some kind, so it's impossible to store the exact value with only 23 binary digits. The fractional part gets cut off and you get a number close to the actual value of `2 / 454` but not the exact number. Once that has happened you end up with results like `.9999999` because your `f` is not exactly `1 / 227` when you multiply it by `227`.

Jack's suggestion is a good one and might avoid all the error. The evaluation order matters with floating point because if you do the multiplication first you avoid using the `1 / 227` value, instead you do `227 * 2` which is easily stored as an exact number in the 23 binary digits of precision.
Reply


Messages In This Thread
Possible floating point error? - by TerryRitchie - 05-30-2024, 01:02 AM
RE: Possible floating point error? - by Jack - 05-30-2024, 01:34 AM
RE: Possible floating point error? - by JRace - 05-30-2024, 01:48 AM
RE: Possible floating point error? - by DSMan195276 - 05-30-2024, 01:49 AM
RE: Possible floating point error? - by SMcNeill - 05-30-2024, 02:30 AM
RE: Possible floating point error? - by SMcNeill - 05-30-2024, 03:31 AM



Users browsing this thread: 4 Guest(s)