  We present a matrix factorization algorithm that scales to input matrices
that are large in both dimensions (i.e., that contains morethan 1TB of data).
The algorithm streams the matrix columns while subsampling them, resulting in
low complexity per iteration andreasonable memory footprint. In contrast to
previous online matrix factorization methods, our approach relies on
low-dimensional statistics from past iterates to control the extra variance
introduced by subsampling. We present a convergence analysis that guarantees us
to reach a stationary point of the problem. Large speed-ups can be obtained
compared to previous online algorithms that do not perform subsampling, thanks
to the feature redundancy that often exists in high-dimensional settings.
