  Knowing the effect of an intervention is critical for human decision-making,
but current approaches for causal effect estimation rely on manual data
collection and structuring, regardless of the causal assumptions. This
increases both the cost and time-to-completion for studies. We show how large,
diverse observational text data can be mined with large language models (LLMs)
to produce inexpensive causal effect estimates under appropriate causal
assumptions. We introduce NATURAL, a novel family of causal effect estimators
built with LLMs that operate over datasets of unstructured text. Our estimators
use LLM conditional distributions (over variables of interest, given the text
data) to assist in the computation of classical estimators of causal effect. We
overcome a number of technical challenges to realize this idea, such as
automating data curation and using LLMs to impute missing information. We
prepare six (two synthetic and four real) observational datasets, paired with
corresponding ground truth in the form of randomized trials, which we used to
systematically evaluate each step of our pipeline. NATURAL estimators
demonstrate remarkable performance, yielding causal effect estimates that fall
within 3 percentage points of their ground truth counterparts, including on
real-world Phase 3/4 clinical trials. Our results suggest that unstructured
text data is a rich source of causal effect information, and NATURAL is a first
step towards an automated pipeline to tap this resource.
