LLM predictions match human forecasters for social science experiments, but overestimate effect sizes
Can a large language model predict the outcome of a social science experiment before it is run? According to a study published July 8 in Nature by Ashwini Ashokkumar (Harvard), Luke Hewitt (Stanford/Transluce), Isaias Ghezae (Harvard), and Robb Willer (Stanford), the answer is yes, up to a point.
GPT-4 predicted treatment effects across 70 preregistered, nationally representative U.S. survey experiments encompassing 469 treatment effects and 119,330 human participants, achieving accuracy comparable to pooled human forecasters. The key limitation: the model systematically overestimated effect sizes by roughly 80%, requiring calibration before its predictions became practically useful.
What the study did
The researchers assembled a primary archive of 70 preregistered experiments from the NSF-funded Time-Sharing Experiments in the Social Sciences (TESS) program, spanning political science, psychology, sociology, social policy, public health, and communication. The experiments tested a wide range of interventions: framing effects, salience manipulations, priming of social identities, and others, measuring outcomes from political attitudes to prejudice to happiness.
For each of the 469 treatment effects, GPT-4 was prompted with the experiment’s design, treatment, and outcome measures, and asked to predict the expected effect size. The same predictions were also made by 460 social scientists, providing a human benchmark.
The results:
- Correlation with actual effects: r = 0.85 (disattenuated r_adj = 0.91), matching pooled human forecasters
- For experiments published after GPT-4’s training cutoff: r = 0.90, ruling out simple memorization
- Open-weight models achieved similar accuracy, further confirming the finding is not GPT-4-specific
- A simple unweighted average of human and LLM predictions yielded even higher accuracy (r = 0.88), suggesting the two sources of prediction are partially complementary
The calibration problem
The high correlation tells only part of the story. Raw GPT-4 predictions had a root mean squared error (RMSE) of 10.9 percentage points, worse than human forecasters at 8.4 percentage points, because the model systematically inflated effect magnitudes.
To correct this, the researchers applied a linear rescaling factor of approximately 0.56, meaning GPT-4’s predicted effects were, on average, about 1.8 times too large. After rescaling, the RMSE dropped to 5.3 percentage points, now better than humans alone (6.0 percentage points) and close to the human+LLM combination (4.7 percentage points).
The overestimation is systematic: GPT-4 correctly detects the direction and relative ordering of effects (hence the high correlation), but inflates absolute magnitudes. The paper cannot pinpoint why, possibilities include training on clear-cut causal narratives, or limits in the model’s ability to aggregate probabilistic knowledge.
Implications for social science
The authors frame the finding as a tool for augmenting, not replacing, traditional experimental methods. An LLM that can rapidly predict which interventions are likely to work, and flag implausible results before expensive data collection, could accelerate the iterative process of hypothesis testing.
The paper also introduces the concept of LLMs as “computational laboratories for virtual experimentation,” where researchers can explore intervention effects across simulated populations before committing resources to field or survey experiments.
The study includes several important caveats. All experiments were conducted on U.S. nationally representative samples; generalizability to other populations is untested. Accuracy was lower for large-scale field experiments (a secondary archive of 15 megastudies with 606 effects), suggesting context matters. And there is a risk of “illusions of understanding”, that researchers may over-rely on LLM predictions without human validation, particularly for underrepresented groups where the model’s training data may be thinner or biased.
For now, the study suggests the highest accuracy comes from combining human and machine predictions. “Scientists should see LLMs as a highly knowledgeable but mildly delusional collaborator,” the authors note, one whose insights are valuable, but whose confidence must be taken with a grain of salt.
Sources:
1. Ashokkumar, A., Hewitt, L., Ghezae, I. & Willer, R. “Large language models can predict the results of social science experiments.” Nature (2026). DOI: 10.1038/s41586-026-10742-x
2. Code Ocean capsule: https://codeocean.com/capsule/9843791/tree/v1
3. Interactive demo: https://treatmenteffect.app/