  Recent advances in the area of long document matching have primarily focused
on using transformer-based models for long document encoding and matching.
There are two primary challenges associated with these models. Firstly, the
performance gain provided by transformer-based models comes at a steep cost -
both in terms of the required training time and the resource (memory and
energy) consumption. The second major limitation is their inability to handle
more than a pre-defined input token length at a time. In this work, we
empirically demonstrate the effectiveness of simple neural models (such as
feed-forward networks, and CNNs) and simple embeddings (like GloVe, and
Paragraph Vector) over transformer-based models on the task of document
matching. We show that simple models outperform the more complex BERT-based
models while taking significantly less training time, energy, and memory. The
simple models are also more robust to variations in document length and text
perturbations.
