4 Things Healthcare Providers Should Know About P-Values in Research
Key takeaways:
P-values are one of the most commonly used — and misunderstood — parts of statistical testing.
They are the probability of an observed result, or a more extreme result, given the null hypothesis is true.
While p-values are useful tools for hypothesis testing, healthcare professionals must take other factors into account when interpreting them.
Let’s face it: Statistics is a difficult subject and can be confusing for even the most experienced healthcare providers.
Even so, statistics provides the foundation for evidence-based medicine. Because of this, having a good understanding of the most common statistical tools used in clinical research is invaluable in providing care to our patients.
The p-value is perhaps the most misunderstood statistical concept used in the analysis and reporting of medical research. To help us better understand it, we interviewed our GoodRx Research team, an expert crew consisting of public health professionals, health economists, and data analysts.
Instant Rx savings for insured and uninsured patients
GoodRx for HCPs has savings that can beat insurance copays and reduce your need to complete prior authorizations.
GoodRx is NOT insurance. GoodRx Health information and resources are reviewed by our editorial staff with medical and healthcare policy and pricing experience. See our editorial policy for more detail. We also provide access to services offered by GoodRx and our partners when we think these services might be useful to our visitors. We may receive compensation when a user decides to leverage these services, but making them available does not influence the medical content our editorial staff provides.
Here are their top 4 things to know about p-values.
1. What a p-value is (and what it’s not)
Let’s start with the definition of a p-value and common misinterpretations of p-values.
The definition of a p-value
The American Statistical Association defines a p-value as “the probability under a specified statistical model that a statistical summary (e.g. the sample mean difference between two compared groups) would be equal to or more extreme than its observed value.”
According to the GoodRx Research team, a p-value is the chance of observing the result you found, or a more extreme result, given that the null hypothesis is true. In medical research, the null hypothesis is usually that there is no effect with the drug or treatment being studied.
In other words, if the null hypothesis is true, then under repeated trials of the experiment, it is the percentage of times the data you get would be equal to, or more extreme than, the data observed.
Misinterpretations and reproducibility
The GoodRx Research team notes that a common misinterpretation is that a p-value is the probability that the null hypothesis is true.
However, a p-value is actually a probability calculated based on the assumption that the null hypothesis is true. This difference is important because of concerns with reproducibility of trial results.
As the American Statistical Association (ASA) notes: “The smaller the p-value, the greater the statistical incompatibility of the data with the null hypothesis, if the underlying assumptions used to calculate the p-value hold. This incompatibility can be interpreted as casting doubt on or providing evidence against the null hypothesis or the underlying assumptions.”
In other words, it is evidence against a hypothesis, not a source of truth.
2. Beware of statistical fallacies
The GoodRx Research team also points out that it is equally important to understand the limitations of p-values as well as some of the current problems with their use.
Publication bias
Typically, researchers have pre-defined research questions and hypotheses to explore. In “p-hacking,” they investigate data for ad-hoc research questions until significant results are found. And then, only those results are reported.
Another concern is with publication bias, where journals tend to publish far more trials with positive results than negative results – this leads to p-hacking among researchers who want their work to be published.
Statistical significance
Another key point from the ASA is that conclusions should not be based solely on whether or not a p-value passes a specific threshold, or what we often call statistical significance. A conclusion does not automatically become “true” or “false” after it passes a particular threshold. Cherry-picking or only presenting those data, like p<0.05, prevents proper interpretation of findings.
Healthcare providers should consider other factors as well, such as study design, quality of the measurements, and external evidence for what is being studied. The GoodRx Research team recommends asking yourself whether or not the results align with existing studies, whether they make sense, and whether they are meaningful beyond just passing a statistical threshold.
Statisticians have actually argued for abandoning the phrase “statistical significance,” eliminating arbitrary cutoffs like p<0.05, and accepting that uncertainty exists in all research. That is perhaps the best way to think about any clinical trial result, whether it includes p-values or not.
3. P-values do not measure effect size
P-values only measure the statistical evidence for or against a hypothesis. They do not measure the clinical significance of the results.
A smaller p-value does not imply a larger effect size, and in fact does not measure effect size at all. On the flip side, a large p-value does not necessarily imply no effect. As the ASA notes, small sample sizes or imprecise measurements can produce large p-values, even if there is a treatment effect.
Study results need to be taken into context. Ask yourself if your patient would truly benefit, assuming they responded to the treatment in the same way as what was reported.
4. Know where and when to ask for help
Even statisticians need help from other statisticians sometimes. So it’s only natural that healthcare professionals who aren’t statisticians might need help interpreting statistics used in research.
Many health systems hire statisticians. And while they might have been hired to help analyze internal hospital data and improve quality and performance, they can be invaluable in understanding new evidence.
Consider asking the department if they are willing to have one of their statisticians join the Pharmacy and Therapeutics (P and T) Committee, for example. A hospital journal club that involves students, residents, attending providers, pharmacists, and a statistician is another way to include a resident expert in the learning process.
P-value pitfalls in practice
A study published in BMC Medicine in 2018 highlights the importance of p-values and reproducibility in practice. The authors examined 275 articles in critical care, covering 158 practices, and published in The New England Journal of Medicine, The Lancet, or JAMA. Out of those 158 practices, they found 66 practices where a later trial attempted to reproduce the original trial results.
They found that the original study generally reported larger effect sizes than the reproduction attempt. And over half of all clinical practices investigated had results in the reproduction that were inconsistent with the original trial.
They even found two clinical practices that were reported effective in the original trial but reported to be harmful in the reproduction. One of those practices is tight glycemic control, a classic example where early, observational trials and one randomized clinical trial in a limited subset of critically ill patients led to widespread adoption. They reported a p-value for the improvements in mortality to be 0.005, far below the 0.05 threshold that is widely used.
Eight years after that initial trial was published, another randomized controlled trial reported a 14% increase in the odds of death from tight glycemic control compared to less intensive management. Today, most medical societies recommend less intensive control.
The bottom line
P-values, when used correctly and interpreted reasonably, can provide valuable insights into the compatibility of clinical trial results with a hypothesis. Widespread misinterpretations of p-values have led to concerns from the statistics community. Understanding how best to interpret p-values can help providers better understand the risks and benefits of treatment options for their patients so they can provide their best care.
Why trust our experts?
