This is not a joke nor a teasing title! Medical studies, humanities, business research, psychology, etc … everybody is concerned! Actually, it’s a very serious issue about the flaws of P-value used as a scientific objective mean for rejecting the null hypothesis. This kind of statistical validation is being held in all scientific fields as the mainstream way to establish a rule or a scientific law or make a result accepted and published. The way we report our experiments via P-value gives as the false illusion of precision and validity!
In a famous paper published by Loannidis in 2005 (Why Most Published Research Findings Are False) the author conducted simulations proving that most scientific results are more likely to be false than true. The chase of statistical significance is not a warranty that the results are true. According to the author the bias could be coming from many elements such as : weak effects, experimental design flexbility, sample sizes, etc… Obtaining a significant P-value can be the result of chance and not the evidence of an effect (the opposite is also true, where research can be unable to observe a true hidden effect).
Pr. Geoff Cumming defends the idea of a « New statistics » way of working to ensure that we report in research the confidence intervals instead of the misleading P-value. The above video is a simulation explaining how P-value is also in someway a kind of -random metric- depending on other parameters! and that the only way for science to advance is to express results as an estimation or likelihood confidence interval to compare.
Researchers should conduct many experiments and encourage metanalysis that aim to make those confidence intervals smaller (more precise) by replication and comparing if those effects can validate (or invalidate) our expectations or research conjectures.
This approach will reset the way we do research by creating incentives to search for the cummulative truth, instead of looking for an artificially probably biaised P-value. It’s also a more humble, honnest and precise way to share scientific research findings and to boost the cummulative aspect of science instead of seeking unreliable novelty or working with a silo research mindset. Our P-values gives us very poor information and exposes heavily research to the danger of false inferences.
Ressources to consult :
www.thenewstatistics.com : The excellent website of Pr. Cumming, you can download here ESCI software (Excel files) that allow you to run experiments and simulations manually & discover how P-value can be tricky and misinform research.
Buy or rent the book : Cumming, Geoff. Understanding the new statistics: Effect sizes, confidence intervals, and meta-analysis. Routledge, 2013. (A new version will be published in August 2016)
Author: Yassine El Bouchikhi