on my machine i am seeing 1
Code: Select all
dim as ulong a = 4294967282
dim as single s = a / 2^32
? s, a; " "; culng(-1)
Code: Select all
dim as ulong a = 4294967282
dim as single s = a / 2^32
? s, a; " "; culng(-1)
That result is OK, since Single doesn't posses the necessary amount of precision, for such small differences.dafhi wrote:num / (num+1) supposed to be less than 1 ... on my machine i am seeing 1
Code: Select all
dim as single s = a / (2^32 + 128)