  Although much effort has recently been devoted to training high-quality
sentence embeddings, we still have a poor understanding of what they are
capturing. "Downstream" tasks, often based on sentence classification, are
commonly used to evaluate the quality of sentence representations. The
complexity of the tasks makes it however difficult to infer what kind of
information is present in the representations. We introduce here 10 probing
tasks designed to capture simple linguistic features of sentences, and we use
them to study embeddings generated by three different encoders trained in eight
distinct ways, uncovering intriguing properties of both encoders and training
methods.
