10-12-2023, 09:48 AM
The program calculates the probability for data stored in a 2 dimensional array 112,13.
Five digits to the right of the decimal is what I am using but can change if necessary to
improve the overall function of the program.
The problem is not with the N2S$ function as I have a test program so that I can look
up the values from the array for each calculation, when I plug them in directly it outputs
the correct formatted value.
I have changed the data type for the array to Single, double, _float but nothing helps.
The program works by cycling through the array pulling out two values then calculates
the probability, simple. I write the output values to a file making it easy to test. The
probabilities are also stored in another array for the next stage of processing.
The function was added to my main program, Like I said it pulls out two values from the
array then simply divides them to get a probability. This probability is then copied to a
string A$=Ltrim$(Str$(A)) then ran through the N2S$ function to check for scientific notation
and if so then convert it. The string is then adjusted to a preset length. A$=Mid$(A$,1,6), etc.
Like I said the N2S$ works perfect when I use the test tool but fails in the body of the main
code. Both are almost identical as far as the calculations, one works the other, not so much.
I am processing millions of cycles overall so speed is a factor.
Any thoughts/ideas are welcome.
R1
Five digits to the right of the decimal is what I am using but can change if necessary to
improve the overall function of the program.
The problem is not with the N2S$ function as I have a test program so that I can look
up the values from the array for each calculation, when I plug them in directly it outputs
the correct formatted value.
I have changed the data type for the array to Single, double, _float but nothing helps.
The program works by cycling through the array pulling out two values then calculates
the probability, simple. I write the output values to a file making it easy to test. The
probabilities are also stored in another array for the next stage of processing.
The function was added to my main program, Like I said it pulls out two values from the
array then simply divides them to get a probability. This probability is then copied to a
string A$=Ltrim$(Str$(A)) then ran through the N2S$ function to check for scientific notation
and if so then convert it. The string is then adjusted to a preset length. A$=Mid$(A$,1,6), etc.
Like I said the N2S$ works perfect when I use the test tool but fails in the body of the main
code. Both are almost identical as far as the calculations, one works the other, not so much.
I am processing millions of cycles overall so speed is a factor.
Any thoughts/ideas are welcome.
R1