  The development of powerful natural language models have increased the
ability to learn meaningful representations of protein sequences. In addition,
advances in high-throughput mutagenesis, directed evolution, and
next-generation sequencing have allowed for the accumulation of large amounts
of labeled fitness data. Leveraging these two trends, we introduce Regularized
Latent Space Optimization (ReLSO), a deep transformer-based autoencoder which
features a highly structured latent space that is trained to jointly generate
sequences as well as predict fitness. Through regularized prediction heads,
ReLSO introduces a powerful protein sequence encoder and novel approach for
efficient fitness landscape traversal. Using ReLSO, we explicitly model the
sequence-function landscape of large labeled datasets and generate new
molecules by optimizing within the latent space using gradient-based methods.
We evaluate this approach on several publicly-available protein datasets,
including variant sets of anti-ranibizumab and GFP. We observe a greater
sequence optimization efficiency (increase in fitness per optimization step) by
ReLSO compared to other approaches, where ReLSO more robustly generates
high-fitness sequences. Furthermore, the attention-based relationships learned
by the jointly-trained ReLSO models provides a potential avenue towards
sequence-level fitness attribution information.
