10-13-2023, 10:50 AM
@random1
binary floating point is imprecise especially when converting from/to decimal
if the floating point integer+fraction is a sum of powers of 2 then all is well up to the limit of precision, but anything else is approximate, just like 1/3 can't be represented exactly in decimal .1 can't be represented exactly in binary floating point
binary floating point is imprecise especially when converting from/to decimal
if the floating point integer+fraction is a sum of powers of 2 then all is well up to the limit of precision, but anything else is approximate, just like 1/3 can't be represented exactly in decimal .1 can't be represented exactly in binary floating point