  Pre-trained contextualized language models such as BERT have shown great
effectiveness in a wide range of downstream Natural Language Processing (NLP)
tasks. However, the effective representations offered by the models target at
each token inside a sequence rather than each sequence and the fine-tuning step
involves the input of both sequences at one time, leading to unsatisfying
representations of various sequences with different granularities. Especially,
as sentence-level representations taken as the full training context in these
models, there comes inferior performance on lower-level linguistic units
(phrases and words). In this work, we present BURT (BERT inspired Universal
Representation from Twin Structure) that is capable of generating universal,
fixed-size representations for input sequences of any granularity, i.e., words,
phrases, and sentences, using a large scale of natural language inference and
paraphrase data with multiple training objectives. Our proposed BURT adopts the
Siamese network, learning sentence-level representations from natural language
inference dataset and word/phrase-level representations from paraphrasing
dataset, respectively. We evaluate BURT across different granularities of text
similarity tasks, including STS tasks, SemEval2013 Task 5(a) and some commonly
used word similarity tasks, where BURT substantially outperforms other
representation models on sentence-level datasets and achieves significant
improvements in word/phrase-level representation.
