How is statistical analysis like the Odyssey? Here’s an analogy that I used in my research methods course last semester to explain the purpose of study pre-registration. If you’ve ever read the Odyssey, you’ll recall the story of the Sirens, the enchanting lady-monsters whose singing lures to certain death any sailor who hears them. (See Wikipedia for crib notes.) On the advise of his witch-friend Circe, Odysseus pulls a stunt so that he can hear the song of the Sirens while still making it safely past. He instructs his crew to plug their ears with beeswax and then lash him to the mast of his boat. As they sail past the Sirens, Odysseus hears the beautiful voices come-hithering and begs his men to free him, but they tie him up tighter until they are all safely out of ear-shot.
I think this is a good analogy for the benefits of pre-registering your experiments. Statistical significance testing is an alluring thing. It provides us, as data-analysts, with a way of drawing a definitive conclusion about whatever phenomenon we’re studying—“This effect is non-zero!”—and thus to write compelling articles and get published and get tenure and so on. But we also now know that statistical significance testing is easily abused. Using flexible data analysis procedures, it is easy to obtain statistically significant results from totally meaningless data. And statistical significance is so alluring, why should any scholar believe that you haven’t hacked your way to get there? Is there any way to conduct hypothesis tests and actually believe (and convince others) that you’ve ruled out a null at the end of it?
That’s where study pre-registration comes in. Pre-registration involves creating a public record of the exact plans you intend to follow when collecting and analyzing data, in advance of conducting the study. It is like tying your arms and legs to the mast of your ship as you sail through the straights of data collection and analysis. No matter how tempting it is to control for a couple of other covariates…no matter how much cleaner that log-transformation looks…your pre-registered protocol keeps you tied down, preventing you from throwing yourself overboard into the sea of questionable research practices. And as you come out the other side, you can actually put stock in your findings, having heard the siren song of statistical significance and lived to tell the tale.
Back to top