Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Officially goin out of my mind. Trivial INT problem
#9
(10-31-2022, 05:04 PM)bplus Wrote: My sympathies to Bert, this is so not intuitive! Why y not 35.5100000000000000? Yikes!

Because computers keep track of numbers in BASE 2 binary format.

Here's an exercise for you:  Give me the binary value that represents 0.1.  If you can do that, you'll see what the issue is, forevermore.

Powers of 2...
16  (let's just start with 2 ^ 4)  -- too large
8 (2 ^ 3) -- too large
4 (2 ^ 2) -- too large
2 (2 ^ 1) -- too large
1 (2 ^ 0) -- too large
.5 (2 ^ -1) -- too large (for 0.1, it is. Tongue )
.25 (2 ^ -2) -- too large
.125 (2 ^ -3) -- too large
.0625 (2 ^ -4) -- TOO SMALL!!

WTH??  We just went from too large a value to represent 0.1, to too small a value!  So how do we fix this discrepancy??  Surly we aren't going to settle for 0.0625 as being "close enough" to represent 0.1!!  Are we??

Of course not!  We continue on, adding smaller halves to our value to try and get as close to 0.1 as possible!

0.03125 (2 ^ -5) -- too small, but when added to 0.0625, we get:  0.0935 -- that's a lot closer to 0.1 than just 2 ^ -4 by itself.

0.00011&B is much closer to 0.1 than 0.0001&B -- but let's keep on adding halves!  Maybe we can eventually find the PERFECT value that represents 0.1!!

(And, at this point, I'll leave it up to you to try so that you can learn what the issue is and why binary -- ie FLOATING POINT -- values are imprecise and must be handled with care.)
Reply


Messages In This Thread
RE: Officially goin out of my mind. Trivial INT problem - by SMcNeill - 10-31-2022, 05:21 PM



Users browsing this thread: 2 Guest(s)