  Estimating the effect of intervention from observational data while
accounting for confounding variables is a key task in causal inference.
Oftentimes, the confounders are unobserved, but we have access to large amounts
of additional unstructured data (images, text) that contain valuable proxy
signal about the missing confounders. This paper argues that leveraging this
unstructured data can greatly improve the accuracy of causal effect estimation.
Specifically, we introduce deep multi-modal structural equations, a generative
model for causal effect estimation in which confounders are latent variables
and unstructured data are proxy variables. This model supports multiple
multi-modal proxies (images, text) as well as missing data. We empirically
demonstrate that our approach outperforms existing methods based on propensity
scores and corrects for confounding using unstructured inputs on tasks in
genomics and healthcare. Our methods can potentially support the use of large
amounts of data that were previously not used in causal inference
