Post by bhao on Aug 7, 2008 14:06:24 GMT
i had some free time (reason being somewhat similar to this) and since the quality of the nwn random number generator has been called into question several times now, i thought, i'd do some tests to see how good it really is. we all know how good our gut feeling is to measure these things (link, link or even link), so let's look at some hard numbers instead.
the point of all this, of course, is, to link to it the next time somebody claims that the nwn random number gen sucks.
i am no expert on the matter (but i know one
(who is on vacation atm 
)), so i went with what i know and i googled a lot. i still believe it is pretty reliable. (also, see last paragraph)
first we need some random numbers that are like the random numbers we experience every day while playing. to get as close as possible, i generated the numbers by logging in and typing "!d100" over and over and recording the result. i did this on an empty server, namely dev, which ran at the time the same mod as the live servers and which, of course, runs on the same hardware as the live servers. to get a constant stream of numbers without interruptions, i did this while i was alone on the server. i do realize, that things like the changing weather may have interfered with that somewhat, but then this is happening on live too, so again the way the numbers are gathered has not changed.
i generated 10106 numbers this way.
here is a graph of the result:
first i did a chi2 test:
X2=92.32
degree of freedom=99
Q=0.6693
this which is far away from the lower and upper critical values (check link for tables and/or calculators)
then i did a lattice test:
3d:
the criteria here being, that if you can see any patterns, it's a bad number generator. screen shot only shows one pov, but i looked at it from all sides, and couldn't see any... feel free to load them in gnuplot yourself to check (link to data). an example of a how randu, a really bad generator looks can be found here.
from these results, i think it is clear, that any perceived bias of the nwn random number generator is purely imaginary. even if it exists, the idea that you can notice it in-game, while these tests do not, seems rather absurd.
so, yes. random, really!
otoh, if anyone wants to disprove this, then here is the file with all 10106 numbers. some good ideas on how to do this can be found here or here. now back to our regular scheduled program.
the point of all this, of course, is, to link to it the next time somebody claims that the nwn random number gen sucks.
i am no expert on the matter (but i know one
(who is on vacation atm 
)), so i went with what i know and i googled a lot. i still believe it is pretty reliable. (also, see last paragraph)first we need some random numbers that are like the random numbers we experience every day while playing. to get as close as possible, i generated the numbers by logging in and typing "!d100" over and over and recording the result. i did this on an empty server, namely dev, which ran at the time the same mod as the live servers and which, of course, runs on the same hardware as the live servers. to get a constant stream of numbers without interruptions, i did this while i was alone on the server. i do realize, that things like the changing weather may have interfered with that somewhat, but then this is happening on live too, so again the way the numbers are gathered has not changed.
i generated 10106 numbers this way.
here is a graph of the result:
first i did a chi2 test:
X2=92.32
degree of freedom=99
Q=0.6693
this which is far away from the lower and upper critical values (check link for tables and/or calculators)
then i did a lattice test:
3d:
the criteria here being, that if you can see any patterns, it's a bad number generator. screen shot only shows one pov, but i looked at it from all sides, and couldn't see any... feel free to load them in gnuplot yourself to check (link to data). an example of a how randu, a really bad generator looks can be found here.
from these results, i think it is clear, that any perceived bias of the nwn random number generator is purely imaginary. even if it exists, the idea that you can notice it in-game, while these tests do not, seems rather absurd.
so, yes. random, really!
otoh, if anyone wants to disprove this, then here is the file with all 10106 numbers. some good ideas on how to do this can be found here or here. now back to our regular scheduled program.
