Large language models can predict the results of social science experiments

There is growing interest in how large language models (LLMs) can advance social and behavioral science [1–5]. Prior work has assessed LLMs’ ability to predict survey responses [6–9], but less is known about whether they can predict the outcomes of social science experiments [10], particularly those absent from training data. Here, we built an archive of 70 pre-registered, nationally representative, U.S. survey experiments, involving 469 experimental effects and 119,330 participants. We prompted an LLM to simulate how representative samples of Americans would respond to experimental stimuli, then inferred treatment effects by comparing simulated responses across conditions. Predictions derived from GPT-4, whose training-data cutoff predated the publication of many studies in our archive, were strongly correlated with actual treatment effects, achieving accuracy similar to pooled human forecasts. Correlations remained high for studies not published or publicly posted by the model’s training-data cutoff date, and for predictions from prominent open-weight models. Despite high correlations, predictions systematically overestimated effect sizes. In a secondary archive of 15 megastudies featuring 606 effects, correlations were lower but comparable to pooled expert forecasters. To assess implications for scientific practice, we surveyed 460 social scientists about likely uses and perceived risks, and used our archives to assess several applications (pilot testing, intervention selection, identifying effects needing replication) and risks (bias, misuse). Together, these results suggest LLMs can augment experimental methods in science and practice while raising important considerations for responsible use.