  In this work, we investigate the performance of untrained randomly
initialized encoders in a general class of sequence to sequence models and
compare their performance with that of fully-trained encoders on the task of
abstractive summarization. We hypothesize that random projections of an input
text have enough representational power to encode the hierarchical structure of
sentences and semantics of documents. Using a trained decoder to produce
abstractive text summaries, we empirically demonstrate that architectures with
untrained randomly initialized encoders perform competitively with respect to
the equivalent architectures with fully-trained encoders. We further find that
the capacity of the encoder not only improves overall model generalization but
also closes the performance gap between untrained randomly initialized and
full-trained encoders. To our knowledge, it is the first time that general
sequence to sequence models with attention are assessed for trained and
randomly projected representations on abstractive summarization.
