  We present Banyan, an improved model to learn semantic representations by
inducing explicit structure over data. In contrast to prior approaches using
structure spanning single sentences, Banyan learns by resolving multiple
constituent structures into a shared one explicitly incorporating global
context. Combined with an improved message-passing scheme inspired by Griffin,
Banyan learns significantly better representations, avoids spurious false
negatives with contrastive learning, and drastically improves memory efficiency
in such explicit-structured models. Using the Self-StrAE framework, we show
that Banyan (a) outperforms baselines using sentential structure across various
settings (b) matches or outperforms unstructured baselines like GloVe
(+augmentations) and a RoBERTa medium (+simcse) pre-trained on 100M tokens,
despite having just a handful of (non-embedding) parameters, and (c) also
learns effective representations across several low resource (Asian and
African) languages as measured on SemRel tasks.
