Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
rounding numbers and converting to string (hiding scientific notation)
#10
Almost motivates me to finish that leg of my string math that requires a way to deal with repetends. I'm hoping for something as simple as...

1 / 3 = .3... so take three digits .333, multiply them by 3, and .999 rounds up to 1; so if .3... gets multiplied by 3, the results are calculated round up the last digit of 9 by 1, giving 1.

2 / 3 = .6... so take three digits .666, multiply them by 3, and 1.998 rounds up to 2; so if .6... gets multiplied by 3, the results are calculated round up the last digit of say 1.999999999998 by 2, giving 2.

I just don't know if all the nuances would hold up, like 2 / 3 = .6... and .6... *1.5 * 2 would still equal 2 by this method and it might, but I would still be concerned there could be some other instances I'm not considering that would make it so calculations on my system would not match a precision calculator with whatever algorithms they employ to handle repetends.

Pete

1 + 1 makes bigger 1.
Shoot first and shoot people who ask questions, later.
Reply


Messages In This Thread
RE: rounding numbers and converting to string (hiding scientific notation) - by Pete - 07-29-2022, 07:43 PM

Possibly Related Threads…
Thread Author Replies Views Last Post
Question Converting input from other devices to standard PC gamepad analog input? madscijr 0 167 01-01-2026, 06:39 PM
Last Post: madscijr
Question Experimenting with a "StringList" type for simpler handling of string arrays/lists Heimdall 18 1,188 12-19-2025, 12:51 PM
Last Post: Heimdall
  Sub not Reconizing Dim as String pmackay 18 1,454 10-16-2025, 03:32 PM
Last Post: pmackay
  Illegal string-number conversion Herve 7 760 07-07-2025, 09:53 AM
Last Post: a740g
  need help printing a scaled string in a custom font to a 32-bit image madscijr 9 1,148 07-03-2025, 04:48 PM
Last Post: SMcNeill

Forum Jump:


Users browsing this thread: 1 Guest(s)