  The statistical dependencies which independent component analysis (ICA)
cannot remove often provide rich information beyond the linear independent
components. It would thus be very useful to estimate the dependency structure
from data. While such models have been proposed, they usually concentrated on
higher-order correlations such as energy (square) correlations. Yet, linear
correlations are a most fundamental and informative form of dependency in many
real data sets. Linear correlations are usually completely removed by ICA and
related methods, so they can only be analyzed by developing new methods which
explicitly allow for linearly correlated components. In this paper, we propose
a probabilistic model of linear non-Gaussian components which are allowed to
have both linear and energy correlations. The precision matrix of the linear
components is assumed to be randomly generated by a higher-order process and
explicitly parametrized by a parameter matrix. The estimation of the parameter
matrix is shown to be particularly simple because using score matching, the
objective function is a quadratic form. Using simulations with artificial data,
we demonstrate that the proposed method improves identifiability of
non-Gaussian components by simultaneously learning their correlation structure.
Applications on simulated complex cells with natural image input, as well as
spectrograms of natural audio data show that the method finds new kinds of
dependencies between the components.
