  Modeling distributions of covariates, or density estimation, is a core
challenge in unsupervised learning. However, the majority of work only
considers the joint distribution, which has limited utility in practical
situations. A more general and useful problem is arbitrary conditional density
estimation, which aims to model any possible conditional distribution over a
set of covariates, reflecting the more realistic setting of inference based on
prior knowledge. We propose a novel method, Arbitrary Conditioning with Energy
(ACE), that can simultaneously estimate the distribution $p(\mathbf{x}_u \mid
\mathbf{x}_o)$ for all possible subsets of unobserved features $\mathbf{x}_u$
and observed features $\mathbf{x}_o$. ACE is designed to avoid unnecessary bias
and complexity -- we specify densities with a highly expressive energy function
and reduce the problem to only learning one-dimensional conditionals (from
which more complex distributions can be recovered during inference). This
results in an approach that is both simpler and higher-performing than prior
methods. We show that ACE achieves state-of-the-art for arbitrary conditional
likelihood estimation and data imputation on standard benchmarks.
