  Canonical Correlation Analysis (CCA) models are powerful for studying the
associations between two sets of variables. The canonically correlated
representations, termed \textit{canonical variates} are widely used in
unsupervised learning to analyze unlabeled multi-modal registered datasets.
Despite their success, CCA models may break (or overfit) if the number of
variables in either of the modalities exceeds the number of samples. Moreover,
often a significant fraction of the variables measures modality-specific
information, and thus removing them is beneficial for identifying the
\textit{canonically correlated variates}. Here, we propose $\ell_0$-CCA, a
method for learning correlated representations based on sparse subsets of
variables from two observed modalities. Sparsity is obtained by multiplying the
input variables by stochastic gates, whose parameters are learned together with
the CCA weights via an $\ell_0$-regularized correlation loss. We further
propose $\ell_0$-Deep CCA for solving the problem of non-linear sparse CCA by
modeling the correlated representations using deep nets. We demonstrate the
efficacy of the method using several synthetic and real examples. Most notably,
by gating nuisance input variables, our approach improves the extracted
representations compared to other linear, non-linear and sparse CCA-based
models.
