  Estimating and storing the covariance (or correlation) matrix of
high-dimensional data is computationally challenging because both memory and
computational requirements scale quadratically with the dimension. Fortunately,
high-dimensional covariance matrices as observed in text, click-through,
meta-genomics datasets, etc are often sparse. In this paper, we consider the
problem of efficient sparse estimation of covariance matrices with possibly
trillions of entries. The size of the datasets we target requires the algorithm
to be online, as more than one pass over the data is prohibitive. In this
paper, we propose Active Sampling Count Sketch (ASCS), an online and one-pass
sketching algorithm, that recovers the large entries of the covariance matrix
accurately. Count Sketch (CS), and other sub-linear compressed sensing
algorithms, offer a natural solution to the problem in theory. However, vanilla
CS does not work well in practice due to a low signal-to-noise ratio (SNR). At
the heart of our approach is a novel active sampling strategy that increases
the SNR of classical CS. We demonstrate the practicality of our algorithm with
synthetic data and real-world high dimensional datasets. ASCS significantly
improves over vanilla CS, demonstrating the merit of our active sampling
strategy.
