PodcastsThe Peter Attia Drive#269 - Good vs. bad science: how to read and understand scientific studies

#269 - Good vs. bad science: how to read and understand scientific studies
The Peter Attia DriveSun Sep 03 2023
Types of Studies:
- Experimental studies rely on randomization to ensure unbiased results and include a control group for comparison.
- Observational studies are prone to selection bias and recall bias, which can affect the accuracy of data collection.
Phases of Clinical Trials:
- Phase 1 focuses on determining toxicity levels and dose escalation for new drugs or treatments.
- Phase 2 evaluates safety and efficacy in larger populations, while phase 3 provides evidence for regulatory approval through rigorous trials with large sample sizes and controlled conditions.
- Phase 4 gathers additional data post-approval to evaluate effectiveness, side effects, and broader applications.
Confounding Factors:
- Confounders are variables that may influence the relationship between independent and dependent variables in a study. They need to be accounted for or controlled to avoid biased results.
Primary vs Secondary Outcomes:
- Primary outcomes are predefined measures used as the main endpoint for evaluating treatment efficacy in a study.
- Secondary outcomes are additional measures analyzed alongside primary outcomes but hold less weight.
Multiple Hypothesis Testing Problem:
- Multiple hypothesis testing increases the likelihood of finding significant results by chance alone. Correcting p-values is necessary by dividing them by the number of comparisons made.
Performance Bias:
- Performance bias occurs when participants or investigators modify behavior due to observation or knowledge thereof. This bias should be minimized or addressed in experimental studies.
Blinding:
- Blinding ensures participants and/or investigators remain unaware of treatment allocation, minimizing bias. Single blinding refers to participant ignorance, while double-blinding involves both participants and investigators being blinded.
Blinding in Studying Psychedelics:
- Blinding is challenging when studying psychedelics because it's difficult to blind users due to noticeable effects like psilocybin. The difficulty introduces biases that may affect study results.
Size, Duration, and Generalizability of Studies:
- Study size refers to the number of participants involved, duration describes how long the study lasts, and generalizability concerns whether the study population resembles the target audience. Studies with larger sample sizes, longer durations, and similar populations are generally considered more reliable and applicable.
Strengths and Weaknesses of Mass Heterogeneity of Studies:
- Mass heterogeneity in terms of patient population increases generalizability but makes it harder to find significant results. It provides comprehensive insights but introduces variability and potential confounding factors.
Gender Bias in Clinical Trials:
- Historically, clinical trials have been conducted predominantly on men due to assumptions about hormonal complexities in women. This gender bias raises concerns about applying findings from male-dominated studies to both sexes.
Importance of Understanding Study Design:
- Differentiating multi-site studies from single site ones impacts reliability. Although multi-site studies bring heterogeneity, they also introduce biases from different sites.
- Key factors like power analysis adequacy, measurement techniques used, adverse event frequency/severity/distribution, funding sources, conflicts of interest among authors contribute to evaluating study rigor.
Significance Testing: P-values and Confidence Intervals:
- P-values indicate the probability that observed results occurred by chance alone. Conventionally p-values ≤0.05 are considered statistically significant.
- However, interpreting them alongside confidence intervals and effect sizes is crucial for a comprehensive understanding.
- Confidence intervals provide a range within which the true population parameter is likely based on sample data.
Importance of Pre-registration and Combating Publication Bias:
- Pre-registration requires researchers to submit protocols before conducting experiments or collecting data. It reduces selective reporting and publication bias.
- Registered Reports publishing format provisionally accepts research studies based on protocol quality rather than outcomes, minimizing positive result bias in scientific publications.
Review Process for Journal Publication:
- After manuscript submission, editors assess eligibility, novelty, relevance before inviting relevant experts from their editorial board as reviewers.
- Peer reviewers evaluate methodology, results, conclusions' significance. Reviews can be single-blinded (reviewers know author identity) or double-blinded (both unaware).
- Reviewers recommend acceptance/rejection/revision post feedback addressing flaws, suggesting further experiments if necessary.
- Editors make final decisions considering reviewer comments and revisions.
Impact Factor as an Indicator of Journal Reputation:
- Impact factor measures a journal's prestige based on cited references divided by published articles within a specific period.
- Journals with higher impact factors generally include frequently cited articles in reputable journals like New England Journal of Medicine, Lancet, JAMA.