Caffeine: works in some people
There’s a separate debate about caffeine beyond its main effec — whether taking caffeine is still worth it when you factor in the dependence caffeine produces as a drug.
That question doesn’t interest me, but I’ll explain the logic. When we run studies on people who take caffeine, we generally see lots of mental improvements. Why these happen is not clear: perhaps it is an improvement that tapers off as the drug is cleared from the body:
Or it is reversing a pathology that stems from the body’s adaptation to caffeine:
When effect sizes are compared in people who takes lots of caffeine vs not so much of it, they are not the same. They lean towards the reversal hypothesis: caffeine consumers’ mental alertness and reaction time reverts to baseline when they take caffeine.
The problem here is that these are not randomly assigned groups. People who take lots of caffeine might be taking it because the drug has a stronger effect on them, or because they tolerate it well. Notably, levels of anxiety are unchanged when caffeine-takers take caffeine.
This is a debate that can’t be evaluated cleanly with a small study or even a meta-analysis of them. Cross-sectional, observational evidence that control effectively for demographics/genes would be very useful here — like the type of studies that evaluate the objective effects of ADHD medications on outcomes like car crashes or crime, but a little different. It’s perhaps not even a good question to answer with science: people who do well on caffeine should take it, and people who don’t shouldn’t.
I.
There are two good meta-analyses on the effects of caffeine.
The first is an analysis of the effect of caffeine on healthy, rested adults. Not exactly a representative population, but it will do for now. It found roughly identical effects for both reaction time and accuracy:
Results: Thirty-one trials with a total of 1455 participants were included in the meta-analysis. Significant effects in favor of caffeine were found for both accuracy, g = 0.27, and reaction time, g = 0.28. Subgroup analyses showed that higher doses of caffeine (≥ 200 mg) improved both reaction time and accuracy more than lower doses, but whereas a positive linear dose-response relationship was found for reaction time, a quadratic relationship was found for accuracy. The effect of caffeine was not related to differences in habitual caffeine consumption, task complexity, or which attention network was taxed.
Their supposed quadratic effect is nonsense; the p-value (0.032) of it is frankly is laughable. Pooled together, I think they would find a dose-response effect that tapers off after 200-300mg.
They tested for publication bias, but didn’t find anything statistically significant. Visually, it does look like it exists; the studies with higher effect sizes have smaller sample sizes. The priors for it existing are pretty high.
I’d guess that publication bias is inflating the meta-analytic effect size by 0.1-0.2; I think it’s fair to say caffeine’s effect on cognition in healthy people is in the 0.1-0.3 range at high (200-400mg) doses.
A second meta-analysis assessed the effect of caffeine on treating sleep deprivation, which they defined rather loosely. Averaging all of their effect sizes yields an aggregate effect size of d = 1.08, aka 16 IQ points.
Caffeine is widely used to counteract the effects of sleep loss. This systematic review and meta-analysis examined the impact of acute caffeine consumption on cognitive, physical, occupational and driving performance in sleep deprived/restricted individuals. 45 publications providing 327 effect estimates (EEs) were included in the review. Caffeine improved response time (44 EEs; g = 0.86; 95 % CI: 0.53–0.83) and accuracy (27 EEs; g = 0.68; 95 % CI: 0.48–0.88) on attention tests, improved executive function (38 EEs; g = 0.35; 95 % CI: 0.15–0.55), improved reaction time (12 EEs; g = 1.11; 95 % CI: 0.75–1.47), improved response time (20 EEs; g = 1.95; 95 % CI: 1.39–2.52) and accuracy (34 EEs; g = 0.43; 95 % CI: 0.30–0.55) on information processing tasks, and enhanced lateral (29 EEs; g = 1.67; 95 % CI: 1.32–2.02) and longitudinal (12 EEs; g = 1.60; 95 % CI: 1.16–2.03) measures of vehicular control on driving tests. Studies also typically indicated benefit of caffeine on memory (25 EEs), crystallized intelligence (11 EEs), physical (39 EEs) and occupational (36 EEs) performance. Ingestion of caffeine is an effective counter-measure to the cognitive and physical impairments associated with sleep loss.
I looked at their funnel plots, and it looks like a few of the effect sizes may be inflated by publication bias or small study effects. Among these, I count response speed and maybe vehicular control. That said, I think it’s still likely that the overall effect is high.
It’s been estimated that 30% of American adults are sleep-deprived; I assume severe cases are closer to the 1-10% mark. If that is the case, I’d say the overall effect size of caffeine is 0.24 (0.2*0.95+1.08*0.05).