  Effectively modeling text-rich fresh content such as news articles at
document-level is a challenging problem. To ensure a content-based model
generalize well to a broad range of applications, it is critical to have a
training dataset that is large beyond the scale of human labels while achieving
desired quality. In this work, we address those two challenges by proposing a
novel approach to mine semantically-relevant fresh documents, and their topic
labels, with little human supervision. Meanwhile, we design a multitask model
called NewsEmbed that alternatively trains a contrastive learning with a
multi-label classification to derive a universal document encoder. We show that
the proposed approach can provide billions of high quality organic training
examples and can be naturally extended to multilingual setting where texts in
different languages are encoded in the same semantic space. We experimentally
demonstrate NewsEmbed's competitive performance across multiple natural
language understanding tasks, both supervised and unsupervised.
