  Observational studies of causal effects require adjustment for confounding
factors. In the tabular setting, where these factors are well-defined, separate
random variables, the effect of confounding is well understood. However, in
public policy, ecology, and in medicine, decisions are often made in
non-tabular settings, informed by patterns or objects detected in images (e.g.,
maps, satellite or tomography imagery). Using such imagery for causal inference
presents an opportunity because objects in the image may be related to the
treatment and outcome of interest. In these cases, we rely on the images to
adjust for confounding but observed data do not directly label the existence of
the important objects. Motivated by real-world applications, we formalize this
challenge, how it can be handled, and what conditions are sufficient to
identify and estimate causal effects. We analyze finite-sample performance
using simulation experiments, estimating effects using a propensity adjustment
algorithm that employs a machine learning model to estimate the image
confounding. Our experiments also examine sensitivity to misspecification of
the image pattern mechanism. Finally, we use our methodology to estimate the
effects of policy interventions on poverty in African communities from
satellite imagery.
