workings of STR

General FreeBASIC programming questions.
Post Reply
BasicCoder2
Posts: 3906
Joined: Jan 01, 2009 7:03
Location: Australia

workings of STR

Post by BasicCoder2 »

Why do I get different results?

Code: Select all

dim as single number
for i as integer = 0 to 19
    number = int(rnd(1)*10000)/100
    print str(number)
    'print str(int(rnd(1)*10000)/100)
next i
sleep
Here 97.01 becomes 97.0100000000001

Code: Select all

'dim as single number
for i as integer = 0 to 19
    'number = int(rnd(1)*10000)/100
   ' print str(number)
    print str(int(rnd(1)*10000)/100)
next i
sleep
bcohio2001
Posts: 556
Joined: Mar 10, 2007 15:44
Location: Ohio, USA
Contact:

Re: workings of STR

Post by bcohio2001 »

Not sure, but in the first example, you are using a 'single' precision number.

In the second, with no limitations, the division defaults to a 'double'.
counting_pine
Site Admin
Posts: 6323
Joined: Jul 05, 2005 17:32
Location: Manchester, Lancs

Re: workings of STR

Post by counting_pine »

It will be mostly because of the fact that most decimal numbers cannot be represented exactly in binary, in much the same way as thirds/ninths/etc can't be represented exactly in decimal, and as a result there is a small imprecision.

Sometimes (not always) this imprecision will show when the number is printed.
In many cases it is possible to see what lies beyond the digits usually printed, by subtracting by an exact number to drop some significant digits from the start and print more at the end:

Code: Select all

print 9701/100             '' 97.01000000000001
print 9701/100 - 97        ''  0.01000000000000512
print csng(9701/100)       '' 97.01
print csng(9701/100) - 97  ''  0.01000214
(97.01... is a good example here, because we can easily lose three significant digits from the start by subtracting 97.0, which is easy to represent precisely in a precision binary number.)
Post Reply