What NOT to do with NON-“null” results – Part III: Underpowered study, but significant result
What NOT to do with NON-“null” results Part III: Underpowered study, but significant results PREMISE I continue the series on how to interpret experiments that do not go according to plan. In Part I and Part II I alluded to an important but often overlooked issue in research: when you have an underpowered study but a statistically significant result. If this happens a typical response may be: Hooray!!! I did it, even with N = 10 I found the effect I expected! I proved all the naysayers wrong! And I saved a bunch of time and money by not having to collect all the data that annoying “power analysis” spat out! And to top it off, my effect is massive! Way bigger than the literature suggested! Nature, here I come! While you may believe this is cause for celebration, unfortunately, things are not as they seem. Unlike before, this time I say what you shouldn’t do is report “ there is an effect of X, p<.05 ”. While you did reach that coveted statistical significance threshold, yo