Paper Rejected (p>0.05): An Introduction to the Debate on Appropriateness of Null-Hypothesis Testing

Key Problems With P-Based Statistics

The debate on null-hypothesis testing has identified many “sins” of null-hypothesis significance testing (NHST) and the way that it is normally used in scientific work. Here we look at them as we perceive the severity of the problem in mobile-HCI:

• 1.

  Treating NHT as a binary approval of result validity;

• 2.

  Confusing strength of p-value results with effect size;

• 3.

  Abusing the statistical tests themselves;

• 4.

  Making conclusions from non-significant results;

• 5.

  Making illogical arguments based on results.

Reviewing recent proceedings of MobileHCI, we are not as guilty as other domains in which null-hypothesis testing has been criticised. However, we tend to be guilty of the first three sins quite widely and we perceive a risk that as publication becomes more competitive, reviewers might push us further along the route of inappropriate statistics.