  We introduce the Conditional Independence Regression CovariancE (CIRCE), a
measure of conditional independence for multivariate continuous-valued
variables. CIRCE applies as a regularizer in settings where we wish to learn
neural features $\varphi(X)$ of data $X$ to estimate a target $Y$, while being
conditionally independent of a distractor $Z$ given $Y$. Both $Z$ and $Y$ are
assumed to be continuous-valued but relatively low dimensional, whereas $X$ and
its features may be complex and high dimensional. Relevant settings include
domain-invariant learning, fairness, and causal learning. The procedure
requires just a single ridge regression from $Y$ to kernelized features of $Z$,
which can be done in advance. It is then only necessary to enforce independence
of $\varphi(X)$ from residuals of this regression, which is possible with
attractive estimation properties and consistency guarantees. By contrast,
earlier measures of conditional feature dependence require multiple regressions
for each step of feature learning, resulting in more severe bias and variance,
and greater computational cost. When sufficiently rich features are used, we
establish that CIRCE is zero if and only if $\varphi(X) \perp \!\!\! \perp Z
\mid Y$. In experiments, we show superior performance to previous methods on
challenging benchmarks, including learning conditionally invariant image
features.
