Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Officially goin out of my mind. Trivial INT problem
#9
(10-31-2022, 05:04 PM)bplus Wrote: My sympathies to Bert, this is so not intuitive! Why y not 35.5100000000000000? Yikes!

Because computers keep track of numbers in BASE 2 binary format.

Here's an exercise for you:  Give me the binary value that represents 0.1.  If you can do that, you'll see what the issue is, forevermore.

Powers of 2...
16  (let's just start with 2 ^ 4)  -- too large
8 (2 ^ 3) -- too large
4 (2 ^ 2) -- too large
2 (2 ^ 1) -- too large
1 (2 ^ 0) -- too large
.5 (2 ^ -1) -- too large (for 0.1, it is. Tongue )
.25 (2 ^ -2) -- too large
.125 (2 ^ -3) -- too large
.0625 (2 ^ -4) -- TOO SMALL!!

WTH??  We just went from too large a value to represent 0.1, to too small a value!  So how do we fix this discrepancy??  Surly we aren't going to settle for 0.0625 as being "close enough" to represent 0.1!!  Are we??

Of course not!  We continue on, adding smaller halves to our value to try and get as close to 0.1 as possible!

0.03125 (2 ^ -5) -- too small, but when added to 0.0625, we get:  0.0935 -- that's a lot closer to 0.1 than just 2 ^ -4 by itself.

0.00011&B is much closer to 0.1 than 0.0001&B -- but let's keep on adding halves!  Maybe we can eventually find the PERFECT value that represents 0.1!!

(And, at this point, I'll leave it up to you to try so that you can learn what the issue is and why binary -- ie FLOATING POINT -- values are imprecise and must be handled with care.)
Reply


Messages In This Thread
RE: Officially goin out of my mind. Trivial INT problem - by SMcNeill - 10-31-2022, 05:21 PM

Possibly Related Threads…
Thread Author Replies Views Last Post
  Int ( money ) inc 5 440 11-30-2025, 12:45 AM
Last Post: hsiangch_ong
  Big problem for me. MystikShadows 11 1,947 01-16-2025, 05:11 AM
Last Post: JRace
  Problem with "MOD" Kernelpanic 5 1,008 01-06-2025, 02:37 AM
Last Post: Pete
  Problem with _Resize dano 7 1,234 10-22-2024, 07:22 PM
Last Post: dano
  Menu Run (No EXE) Problem GareBear 5 1,150 10-06-2023, 05:41 AM
Last Post: GareBear

Forum Jump:


Users browsing this thread: 1 Guest(s)