A brief foray into stats
OK, I am quite aware that getting very far into stats will cause your brain to start bludgeoning your eyeballs while seeking the emergency exit. Unfortunately, if you’re going to grasp theorycrafting, you’re just going to have to buy stock in the pain relief company of your choice, maybe a couple of lumberyards for the eye-propping toothpicks, and spend a little skullsweat. The good news is, I’m not going to take you into chi-squared and all of that. I’m going to just kind of skirt the edges enough for you to have SOME handle on the subject. Grab your painkiller and caffiene, and here we go. Oh, wait… you don’t need to read ANY of this to be a better priest, or even a better player. It’s purely to get some handle on theorycrafting – to understand why we say some of the things we say. If that’s not your cup of tea, you can go somewhere else — there are some very nice links over there to the right, go try some. Onward…
Statistics is the science of making our guesses as close to “what is” as possible. I think that is the most important thing to keep in mind – we might be wrong. We’re trying to be ‘right’ as often as possible, but some of the time we WILL be wrong. All the tools – the confidence levels and statistical deviation calculations and all that sort of thing – are to help us be not wrong as much as possible.
So why does it matter for the underpinnings of WoW? Quite simply, because Blizzard doesn’t tell us the details. Hey, folks, Blizzard is selling a game. If we knew every detail we’d min-max perfectly and you’d wind up seeing, well, a whole bunch of clones. You get some of that anyway, but there are still variations. A lot of the cloning is because we guesstimate and analyze and guesstimate again as to how much XX affects YY which modifies spell ZZ or weapon AA… sigh. The bottom line is that we use statistics as part of our analysis. OK, enough with why, let’s get into basics.
The first thing we’re going to do is recognize that thanks to the way everything’s presented we don’t do “odds”. Everything – EVERYTHING – is probability. The odds of something happening (or not happening) is immaterial. (lecture about the relationship of odds to probability is hereby ejected from this article. Let your brain have a brief sigh of relief.) But we do need to examine this term “probability”.
Probability is simply the proportion of the time whatever-it-is happens. We’ve a simple example almost everyone knows: miss. The base miss in melee is 5%. The proportion of attacks which come up “miss”, before any other modifiers get into the mix, is 5%. Simple, right?
Oh — that 5% is axiomatic. An axiom is something we assume is always true (it may or may not be, but we’re trusting it to be such). We trust it as such because Blizzard’s spokespeople said so. As it happens some very untrusting souls have hammered it with level 1 and higher characters and come up with the same value enough times that the trust is very, very high — given equal levels of characters and no other modifiers, overall 5% of every melee attack will miss. hmmm, lots of qualifiers in there which can help us understand later issues. Let’s take on a couple of them.
No other modifiers. The more complexity the more wildly things can vary. Now we can eventually account for the modifiers, but first we need the basic “all else being equal” standard. This is why of the tests I’ve done neither can be fully trusted. I’ve got a highpower wand that still gives me some oomph in stamina. I’m two levels above my target which throws some other modifiers into the mix. And that’s just one test of the two.
Over time. This one’s going to be run into quite a bit – it’s the basis of our confidence. Luck happens – good and bad, but it happens. If you only flip a coin three times and it comes up heads all three times, you’d be tempted to believe the coin would almost always come up heads. What we know from math and from lots and lots of experiments, however, is that OVER TIME, or GIVEN SUFFICIENT TESTS, you get real close to an equal number of heads and tails. (Close. Allegedly the head of a US quarter is slightly heavier than the tail, but the difference won’t show up unless/until you are flipping the coin over a million times.) But… who wants to flip millions of times? For that matter, what is SUFFICIENT TESTS? Enter confidence level, confidence interval, and margin of error.
ugh. Confidence interval and confidence level jump into the ‘stock in painkiller’ level really, really fast. So instead of detail, let’s get some general concept covered instead.Confidence level is, basically, how likely it is that this number – plus or minus a certain range – is going to be right. That certain range is often but not always the confidence interval — for our simple levels of caffeine and painkiller we’ll say that it is, but if you’re taking a stats class and say so you’ll fail the question. Ummmm – this latter is similar to but not identical to the ‘margin of error’. And I’ll mention credible interval only enough to say “it’s different” and in this case “we don’t care.” Sorry, let’s try to get back to the basic level.
We can use scary math to say: “If I want to be 95% certain that any number is +/-0.01% of the ‘true’ number I need to do XXXX samples.” The higher the certainty wanted or the tighter the range selected, the more samples I need. I’m not going to give you scary math. Instead…
I promised to make this tolerable for you. Here’s a guideline. If you run 100 identical tests for a percental integer (such as 1% or 6%) in WoW, you can be 90% certain that your answer is within 1% of WoW’s real number. (not 0.99 to 1.01, but 0 to 2) You can shift ONE decimal for each magnitude change in tests run. In other words, if you run 100*100=10,000 identical tests, you can be 99% certain of a percental integer +/-1% OR 90% certain of a tenth place (x.y%) within 1% OR 90% certain of being within +/- 0.1 of the integer percent. (back to my two runs – each is around 3,000 identical tests. You can see how much trust they should be given…)
sigh. For most WoW testers, 90% is acceptable, though 95% is the statistician’s “standard”. For most purposes in the WoW tester’s community, given a hundredth’s place percentile (aka, x.yz% such as 0.04%) we consider it acceptably likely it’s “true” after 100,000 samples. (There are statisticians out there who are screaming to correct my math and practices, by the way – if they want to, they’re welcome. I am not a statistician, I just have to grasp the basics, and it won’t be the first time I’m wildly wrong.)
OK, one last thing before I go. When I say identical, I mean identical – but that’s not always possible. So we the theorycrafters try to identify the differences, and make educated guesses as to whether the difference matters or not. We’re not perfect and we’re not getting paid, but this tends to fall into the category of “fun” for some of us so we try very, very hard to be right anyway.
If you doubt what you’re reading, see if they link to or show the work. What some consider acceptable others consider a nice start but far from conclusive. Eventually a concensus will emerge and you’ll get your “magic numbers” to decide if this stat rating is better than that one and by how much.
Which will last till the next patch. Or WotLK. It will change, eventually.
And some of us think of this as fun. pray for us…