EBM 2.0 Interpretation Principles

Be Savagely Skeptical:

• Assume all positive study findings are false until rigorously proven otherwise through repeated, high quality independent replications.

Ask the right question:

• What is the overall probability that our hypothesis is true given everything we already know including what is likely to be hidden from us?
• Note this is not 1 minus the p value (a common and serious misconception).

Answer the right question:

– Bias

• Use the best of EBM 1.0 principles to always closely review studies for any sources of visible bias and assume invisible bias is present.
• Realise that bias changes the very meaning of the p value calculation
  • Therefore discount findings for the presence of bias and reject them entirely if there is substantial bias.
• Specifically account for bias in calculations (see below)

– Chance & Statistics:

• Abandon “Statistical Significance”
  • End the use of the dichotomous concept of “statistical significance” with arbitrary meaningless thresholds for p-values and confidence intervals
• Use the Bayes Factor approach
  • Use p values as likelihood ratios (aka Bayes Factors) to convert pre-test probabilities (aka prior probability) of a finding being real into post-test probabilities (aka posterior probability), where the study is the “test” and the p value is viewed as the test result and accuracy.
    • See this post for how to calculate
    • Be deliberately conservative when choosing baseline pre-test probabilities. See this post for guidance on choosing a baseline pre-test probability that a clinical hypothesis is true.
• Specifically Account for  bias
  • Effect sizes and derived p values and probabilities should be explicitly adjusted for bias.
    • This currently rarely occurs and is a great failing in our analysis.
    • Further large scale research projects are required in medical fields to more accurately estimate the prevailing levels of bias such that bias adjustments can be made.
  • Where an explicit bias adjustment has not occurred, refer to all calculated effect sizes/p values/probabilities as “bias-distorted” effect sizes/p-values/probabilities. This more appropriately describes the truth and such vernacular will help make the intrinsic presence of bias, front of mind when interpreting results.

Accept & Manage Uncertainty:

• Accept uncertainty over the seductive lure of “positive” or”negative”.
• View evidence as merely incrementally changing levels of uncertainty.
• Use real world cost-benefits of therapy to determine acceptable level of certainty required before practice change.
  • e.g. therapies with low net benefit and high costs need the highest levels of certainty to consider utilising – this can only be derived from supportive evidence that is repeatedly reproducible in high quality independent trials.