  This paper presents a novel approach to measuring statistical dependence
between two random processes (r.p.) using a positive-definite function called
the Normalized Cross Density (NCD). NCD is derived directly from the
probability density functions of two r.p. and constructs a data-dependent
Hilbert space, the Normalized Cross-Density Hilbert Space (NCD-HS). By Mercer's
Theorem, the NCD norm can be decomposed into its eigenspectrum, which we name
the Multivariate Statistical Dependence (MSD) measure, and their sum, the Total
Dependence Measure (TSD). Hence, the NCD-HS eigenfunctions serve as a novel
embedded feature space, suitable for quantifying r.p. statistical dependence.
In order to apply NCD directly to r.p. realizations, we introduce an
architecture with two multiple-output neural networks, a cost function, and an
algorithm named the Functional Maximal Correlation Algorithm (FMCA). With FMCA,
the two networks learn concurrently by approximating each other's outputs,
extending the Alternating Conditional Expectation (ACE) for multivariate
functions. We mathematically prove that FMCA learns the dominant eigenvalues
and eigenfunctions of NCD directly from realizations. Preliminary results with
synthetic data and medium-sized image datasets corroborate the theory.
Different strategies for applying NCD are proposed and discussed, demonstrating
the method's versatility and stability beyond supervised learning.
Specifically, when the two r.p. are high-dimensional real-world images and a
white uniform noise process, FMCA learns factorial codes, i.e., the occurrence
of a code guarantees that a specific training set image was present, which is
important for feature learning.
