  Contrastive learning has been shown to produce generalizable representations
of audio and visual data by maximizing the lower bound on the mutual
information (MI) between different views of an instance. However, obtaining a
tight lower bound requires a sample size exponential in MI and thus a large set
of negative samples. We can incorporate more samples by building a large
queue-based dictionary, but there are theoretical limits to performance
improvements even with a large number of negative samples. We hypothesize that
\textit{random negative sampling} leads to a highly redundant dictionary that
results in suboptimal representations for downstream tasks. In this paper, we
propose an active contrastive learning approach that builds an \textit{actively
sampled} dictionary with diverse and informative items, which improves the
quality of negative samples and improves performances on tasks where there is
high mutual information in the data, e.g., video classification. Our model
achieves state-of-the-art performance on challenging audio and visual
downstream benchmarks including UCF101, HMDB51 and ESC50.\footnote{Code is
available at: \url{https://github.com/yunyikristy/CM-ACC}}
