# Hypothesis Testing

Hypothesis Testing– Imagine that your next job is flying through the universe from large planet to large planet in search of non human alien life. Sounds like an awesome job, yes? Unfortunately, as with every dream job, there is… a manager.

And your manager’s giving you an unfortunately paltry user interface. It only has two buttons: YES there is alien life here, NO there’s no alien life here. There’s no maybe / it depends, no place for comments or hedging. Just YES or NO, that’s all you get. And in a further stroke of villainy, this manager has not given you the budget to search the entire planet You’re only gonna be allowed to get out of your spacecraft, start walking until your oxygen supplies get iffy, and then turn around, walk back, and press one of those two buttons.

So we will be dealing with uncertainty here. Which default action would you like? So I’m asking you: “If you don’t even land on this planet, which button you’re gonna hit?” I’m hearing NO… that’s not the only right answer. This depends on the politics of your space exploration company. Maybe the right thing to do in the absence of information is to press the YES button.

That’s an MBA question! So we’ll actually look at how this pans out for both, but let’s start with NO. So, your default action is NO. What’s your null hypothesis? You’re asking yourself, “If I knew everything about the universe, what would have to be true for the NO button to be a good choice?”

This is not a trick question. This is an easy one. When there is in fact no life on this planet, then pressing NO is the happy choice under full information. Great! And what’s the alternative hypothesis? There is alien life on this planet! I couldn’t have said it better myself, well done. So we’ve got our hypotheses set up.

Thank you very much. So here you go, you get out of your spacecraft, you start walking for three grueling, miserable hours and you observe… no aliens. 🙁 What have you learned that’s interesting? The typical human answer – the incorrect one – would be that what we have learned that’s interesting is that there are no aliens on this three-hour walk.

That is an incorrect answer because of how we framed our decision. We are interested – by legal contract – only in the population with respect to its alien / not-alien status. That is, we are interested only in the whole planet. This walk is boring to us and I asked you what we’ve learned that’s interesting. From this walk, we can’t tell whether we saw no aliens here because there are none on the planet or we saw none because they’re under that other rock and we haven’t turned it over yet.

Give me the correct one-word answer. What have we learned that’s interesting? Nothing! Did you notice that we analyzed data and we correctly learned nothing?! Hmm…? How often do you let yourselves learn nothing when you analyze data statistically? Because if you insist on learning something from every statistical analysis, you will tend to learn something stupid!

So I hope that throughout this course I’m going to make you okay with the idea that when you are framing your decisions in this manner, when you are doing statistical inference learning nothing is a very good thing! You should be okay with that! We need a badge that says, “I analyzed data, I learned nothing and I’m proud of that!” So I’m even going to jump up and down in celebration in this course every time that we correctly learn nothing.

Now I’m going to tell you the big secret of statistical inference. Every Frequentist statistical inference course from your STAT101 to your scariest PhD qualifying exam all boils down to this one sentence. It’s just hidden under some spiky math, of course, but it is always this. You could go and (for homework!) derive the entire discipline. You could do that, or you can hang out with me and I’ll break it down for you a little bit.

But, truly, if you sit and meditate on this sentence, everything else follows from this. Okay, so what is this magical sentence? This incantation? When we do hypothesis testing we are always asking the following:

Does the evidence that we collected make our null hypothesis look ridiculous? Yes or no? It is always this – and only this -every single time.

If you can truly internalize this, you’ve got the whole thing. So how are we answering so far about our evidence? What’s the answer? Yes or no? The answer is so far No.

Now imagine if instead of walking for three hours and seeing no aliens, we started walking we observe this. Supposing that that is an alien and not a pickle, what have we learned that’s interesting here?

We have learned that there are, in fact, aliens on this planet. There *is* alien life here! Because if I told you I have observed this here alien… and I’m still considering the possibility that there is no alien life on this planet you will tell me that you have observed an idiot!

This evidence makes my null hypothesis look ridiculous… and so what do I do when evidence makes my hypothesis look ridiculous? I don’t cling to that nonsense! I get rid of that thing! And because we have cunningly made these two hypotheses so that they span all possibilities, by forcing myself to reject one of them, I’m then cornered into concluding the other.

So I must learn something here. I now form a belief. I begin with no beliefs. I’m completely agnostic. I have an action but I don’t have a belief at the beginning, but then through this process I now have formed a belief. So, if I answer “yes” to my testing question, my result is to reject this ridiculous thing and conclude in favor of the alternative.

My physical action will be to press the YES button. If I answer no to this testing question, well, in your STAT101 class, they teach you this really long Baroque paragraph to write. I’m convinced that that’s just designed to hurt students’ wrists. I would allow my undergraduates to write:

“We have learned nothing.” It seems very sad that we go through all this math and effort and we collect all this evidence and then we learn nothing. What a letdown! Until we remember that we’re not in the business of learning or knowing things.

That’s not the point here. We’re in the business of taking reasonable real-world actions (of making decisions) and so we don’t really care what we think we know. Our endgame is just taking an action and so we have a wonderful insurance policy The default action.

That’s what that thing is for. We have a contract, pretty much, that says: “If I know nothing or I know very little… here’s what I’m gonna *do*: the default action!” Because the whole framing is not about knowing stuff, it’s about “Here I am, comfortably trundling along, doing this thing that I was gonna be doing… let’s have an honest shot at seeing if I should NOT do it.”

No particular reason? Okay, off I go and do the thing that I was gonna do. So we have the NO button pressed over here, the YES button pressed over there. Nice and symmetric. Two different actions. I hope you agree with me that they are sensible in light of the framing and the evidence.

Now, if you’re interested in epistemology (the philosophy of knowledge)… Let me show you how broken this thing is. Have we learned anything if we have answered yes to our testing question? Yes, we have. Have we learned anything if we have answered no to our testing question? No! And if you’re struggling with this, you might be Bayesian. Entirely different philosophical branch.

By construction here, we have learned nothing. So, no, we have not formed any knowledge. Now this should trouble you. Down this branch, we learn something. Down that branch, we don’t. There’s only one kind of knowledge that we can get: only in favor of one of the conclusions not the other… and we picked which one that was gonna be. This is dead on arrival. It was designed for making decisions and for some reason industry and science thought that you can get knowledge out of it and now there’s all kinds of blogs shamefacedly having a panic.

You can enjoy those on the weekend but we here aren’t going to cry over it because we’re going to use this for what it’s intended for and that is: making reasonable decisions. Key message: the game of hypothesis testing is all about determining whether the evidence that we have makes our null hypothesis look ridiculous In order to be able to set up statistical hypotheses, you must be clear on what your default action is. Your default action determines your null hypothesis (and the whole course of the analysis).