+- QB64 Phoenix Edition (https://qb64phoenix.com/forum)
+-- Forum: Chatting and Socializing (https://qb64phoenix.com/forum/forumdisplay.php?fid=11)
+--- Forum: General Discussion (https://qb64phoenix.com/forum/forumdisplay.php?fid=2)
+--- Thread: Integer (math) Single = Single (/showthread.php?tid=2111)
Integer (math) Single = Single - SMcNeill - 10-20-2023
So here's something that was glitching the heck out of one of my programs earlier, that you guys might not be aware of:
Code: (Select All)
$Console:Only
Dim f As _Float, i As _Integer64
f = 1: i = 1: g = 1
For x = 1 To 10
Print x, f,
Locate , 30: Print i
f = f * 10 + x
i = i * 10 + x
Next
Now, if you run that, you end up with the following results:
Code: (Select All)
1 1 1
2 11 11
3 112 112
4 1123 1123
5 11234 11234
6 112345 112345
7 1123456 1123456
8 11234567 11234567
9 112345678 112345680
10 1123456789 1123456768If you see our math here, we basically take the previous number, multiply it by 10, and then add the previous pass value. (times 10 + 1)
But take a close look at what happens to our integer value once we get up past 7 digits -- it screws up!!
The reason??
Single precision values are limited to 8 digits and then they swap over to scientific notation.
The math here does the integer part correctly. 11234567 * 10 = 112345670
But the math with the SINGLE variable x gives us a SINGLE value as a return: 112345670 + 8 = 1.12345678E8
Which we then assign to our integer64 variable so it now becomes 1123456780.
If you want to avoid this glitch, change X from being a SINGLE variable to becoming a DOUBLE (or _FLOAT), which can hold enough digits to do the proper math without giving scientific notation results (which result in rounding).
Just one more thing to tuck under your hat and be aware of when programming and working with numbers larger than you can easily count on your fingers and toes.
