What if most scientific papers & rules were false? #P-value #ABTesting #Marketingthema

par marketingthema

This is not a joke nor a teasing title! Medical studies, humanities, business research, psychology, etc … everybody is concerned! Actually, it’s a very serious issue about the flaws of P-value used as a scientific objective mean for rejecting the null hypothesis. This kind of statistical validation is being held in all scientific fields as the mainstream way to establish a rule or a scientific law or make a result accepted and published. The way we report our experiments via P-value gives as the false illusion of precision and validity!

In a famous paper published by Loannidis in 2005 (Why Most Published Research Findings Are False) the author conducted simulations proving that most scientific results are more likely to be false than true. The chase of statistical significance is not a warranty that the results are true. According to the author the bias could be coming from many elements such as : weak effects, experimental design flexbility, sample sizes, etc… Obtaining a significant P-value can be the result of chance and not the evidence of an effect (the opposite is also true, where research can be unable to observe a true hidden effect).

Pr. Geoff Cumming defends the idea of a « New statistics » way of working to ensure that we report in research  the confidence intervals instead of the misleading P-value. The above video is a simulation explaining how P-value is also in someway a kind of -random metric- depending on other parameters! and that the only way for science to advance is to express results as an estimation or likelihood confidence interval to compare. 

Researchers should conduct many experiments and encourage metanalysis that aim to make those confidence intervals smaller (more precise) by replication and comparing if those effects can validate (or invalidate) our expectations or research conjectures.

This approach will reset the way we do research by creating incentives to search for the cummulative truth, instead of looking for an artificially probably biaised P-value. It’s also a more humble, honnest and precise way to share scientific research findings and to boost the cummulative aspect of science instead of seeking unreliable novelty or working with a silo research mindset. Our P-values gives us very poor information and exposes heavily research to the danger of false inferences. 

Ressources to consult :
www.thenewstatistics.com : The excellent website of Pr. Cumming, you can download here ESCI software (Excel files) that allow you to run experiments and simulations manually & discover how P-value can be tricky and misinform research.
Buy or rent the book : Cumming, Geoff. Understanding the new statistics: Effect sizes, confidence intervals, and meta-analysis. Routledge, 2013. (A new version will be published in August 2016)

Author: Yassine El Bouchikhi

One Trackback to “What if most scientific papers & rules were false? #P-value #ABTesting #Marketingthema”

Laisser un commentaire

Entrez vos coordonnées ci-dessous ou cliquez sur une icône pour vous connecter:

Logo WordPress.com

Vous commentez à l'aide de votre compte WordPress.com. Déconnexion / Changer )

Image Twitter

Vous commentez à l'aide de votre compte Twitter. Déconnexion / Changer )

Photo Facebook

Vous commentez à l'aide de votre compte Facebook. Déconnexion / Changer )

Photo Google+

Vous commentez à l'aide de votre compte Google+. Déconnexion / Changer )

Connexion à %s

%d blogueurs aiment cette page :