Introduction
How should we decide what counts as trustworthy evidence? Scientific rigor is not a single characteristic of a study, but a chain of decisions made from the moment a question is conceived to the point at which findings are communicated to the public.
Errors can occur at every stage: the question may be ill-posed, the design may be incapable of answering it, the measurements may be weak, the analysis may be inappropriate, the interpretation may overreach, and the public-facing communication may become distorted.
In this episode, Dr. David Allison, PhD discusses the deeper methodological issues that shape the field’s conclusions. The discussion moves from the philosophy of scientific inquiry to the practical realities of study design, statistical analysis, interpretation, and dissemination.
Related resources
- Join the Sigma newsletter for free
- Subscribe to Sigma Nutrition Premium
- Enroll in the next cohort of our Applied Nutrition Literacy course
- Differences in Nominal Significance (DINS) Error leads to invalid conclusions
- Monty Hall Problem
- The Checklist Manifesto
- Statistics as Principled Argument
- LinkedIn:
- Related episodes:
- [03:30]Interview start
- [06:17]What is true scientific rigor?
- [10:06]Study design and analysis problems in nutrition
- [12:56]The DINS error
- [14:14]Conflation of heterogeneity in response vs. in outcomes
- [17:31]Misunderstanding of p-values and hypothesis testing
- [27:01]Incorrect labelling of “responders” and “non-responders”
- [34:49]Errors related to analysis of secondary outcomes
- [45:01]How can nutrition science improve as a field?
- [51:30]Key ideas segment (Premium-only)
Guest Information
David Allison, PhD is Chief of Nutrition and Director of the Children’s Nutrition Research Center at Baylor College of Medicine, where he leads research in nutrition, obesity, and the rigor and reproducibility of scientific evidence.
Trained in psychology, with BA and graduate degrees from Vassar College and Hofstra University, he also completed postdoctoral fellowships at Johns Hopkins University School of Medicine and at Columbia University College of Physicians and Surgeons/Saint Luke’s-Roosevelt Hospital.
His work is especially recognized for its focus on statistical reasoning, research methodology, and improving transparency and trustworthiness in health science.
Study Notes
Useful Terminology for this Episode
- Rigor: A broad concept referring to the care, precision, and methodological soundness with which a study is conceived, designed, executed, analyzed, interpreted, and reported.
- Invalidating Error: An error that, if corrected, could change the studyʼs results or conclusions, or would change them. Allison uses this concept to distinguish minor imperfections from more serious flaws that undermine the trustworthiness of a paperʼs central claim.
- Difference in Nominal Significance (DINS): A statistical error in which researchers conclude that two groups differ because one group shows a “statistically significant” within-group change while the other does not. This is invalid because significance in one group and non-significance in another does not itself establish a statistically significant between-group difference.
- Treatment-Response Heterogeneity: The idea that different individuals may respond differently to the same intervention. Variability in observed outcomes is not, by itself, evidence of variability in treatment response.
- Pre-registration: The public documentation of a studyʼs hypotheses, design, outcomes, and analysis plan before results are known.
- Multiple Testing: The practice of conducting many statistical comparisons within the same dataset. As the number of tests increases, so does the likelihood of false-positive findings unless the issue is addressed transparently and thoughtfully.
- Frequentist Inference: A commonly used statistical framework in health sciences in which P values and confidence intervals are used to assess how compatible observed data are with a null hypothesis.
- Transparency: Open and accurate disclosure of what was planned, what was done, and what was found. Allison repeatedly frames transparency as essential because readers cannot properly judge a study if key decisions or limitations are hidden.
