Integer (math) Single = Single

[SMcNeill]

So here's something that was glitching the heck out of one of my programs earlier, that you guys might not be aware of:

Code: (Select All)

$Console:Only



Dim f As _Float, i As _Integer64

f = 1: i = 1: g = 1

For x = 1 To 10

    Print x, f,

    Locate , 30: Print i

    f = f * 10 + x

    i = i * 10 + x

Next

Now, if you run that, you end up with the following results: 

Code: (Select All)

1        1                   1
2        11                  11
3        112                 112
4        1123                1123
5        11234               11234
6        112345              112345
7        1123456             1123456
8        11234567            11234567
9        112345678           112345680
10       1123456789          1123456768

If you see our math here, we basically take the previous number, multiply it by 10, and then add the previous pass value. (times 10 + 1)

But take a close look at what happens to our integer value once we get up past 7 digits -- it screws up!!

The reason??

Single precision values are limited to 8 digits and then they swap over to scientific notation.

The math here does the integer part correctly. 11234567 * 10 = 112345670

But the math with the SINGLE variable x gives us a SINGLE value as a return: 112345670 + 8 = 1.12345678E8

Which we then assign to our integer64 variable so it now becomes 1123456780.

If you want to avoid this glitch, change X from being a SINGLE variable to becoming a DOUBLE (or _FLOAT), which can hold enough digits to do the proper math without giving scientific notation results (which result in rounding).

---

Just one more thing to tuck under your hat and be aware of when programming and working with numbers larger than you can easily count on your fingers and toes.