  This paper presents two simple improvements to the Self-Structuring
AutoEncoder (Self-StrAE). Firstly, we show that including reconstruction to the
vocabulary as an auxiliary objective improves representation quality. Secondly,
we demonstrate that increasing the number of independent channels leads to
significant improvements in embedding quality, while simultaneously reducing
the number of parameters. Surprisingly, we demonstrate that this trend can be
followed to the extreme, even to point of reducing the total number of
non-embedding parameters to seven. Our system can be pre-trained from scratch
with as little as 10M tokens of input data, and proves effective across
English, Spanish and Afrikaans.
