  We introduce Latent Meaning Cells, a deep latent variable model which learns
contextualized representations of words by combining local lexical context and
metadata. Metadata can refer to granular context, such as section type, or to
more global context, such as unique document ids. Reliance on metadata for
contextualized representation learning is apropos in the clinical domain where
text is semi-structured and expresses high variation in topics. We evaluate the
LMC model on the task of zero-shot clinical acronym expansion across three
datasets. The LMC significantly outperforms a diverse set of baselines at a
fraction of the pre-training cost and learns clinically coherent
representations. We demonstrate that not only is metadata itself very helpful
for the task, but that the LMC inference algorithm provides an additional large
benefit.
