  Computing universal distributed representations of sentences is a fundamental
task in natural language processing. We propose ConsSent, a simple yet
surprisingly powerful unsupervised method to learn such representations by
enforcing consistency constraints on sequences of tokens. We consider two
classes of such constraints -- sequences that form a sentence and between two
sequences that form a sentence when merged. We learn sentence encoders by
training them to distinguish between consistent and inconsistent examples, the
latter being generated by randomly perturbing consistent examples in six
different ways. Extensive evaluation on several transfer learning and
linguistic probing tasks shows improved performance over strong unsupervised
and supervised baselines, substantially surpassing them in several cases. Our
best results are achieved by training sentence encoders in a multitask setting
and by an ensemble of encoders trained on the individual tasks.
