  Many NLP models operate over sequences of subword tokens produced by
hand-crafted tokenization rules and heuristic subword induction algorithms. A
simple universal alternative is to represent every computerized text as a
sequence of bytes via UTF-8, obviating the need for an embedding layer since
there are fewer token types (256) than dimensions. Surprisingly, replacing the
ubiquitous embedding layer with one-hot representations of each byte does not
hurt performance; experiments on byte-to-byte machine translation from English
to 10 different languages show a consistent improvement in BLEU, rivaling
character-level and even standard subword-level models. A deeper investigation
reveals that the combination of embeddingless models with decoder-input dropout
amounts to token dropout, which benefits byte-to-byte models in particular.
