  Learned representations at the level of characters, sub-words, words and
sentences, have each contributed to advances in understanding different NLP
tasks and linguistic phenomena. However, learning textual embeddings is costly
as they are tokenization specific and require different models to be trained
for each level of abstraction. We introduce a novel language representation
model which can learn to compress to different levels of abstraction at
different layers of the same model. We apply Nonparametric Variational
Information Bottleneck (NVIB) to stacked Transformer self-attention layers in
the encoder, which encourages an information-theoretic compression of the
representations through the model. We find that the layers within the model
correspond to increasing levels of abstraction and that their representations
are more linguistically informed. Finally, we show that NVIB compression
results in a model which is more robust to adversarial perturbations.
