  The computing cost of transformer self-attention often necessitates breaking
long documents to fit in pretrained models in document ranking tasks. In this
paper, we design Query-Directed Sparse attention that induces IR-axiomatic
structures in transformer self-attention. Our model, QDS-Transformer, enforces
the principle properties desired in ranking: local contextualization,
hierarchical representation, and query-oriented proximity matching, while it
also enjoys efficiency from sparsity. Experiments on one fully supervised and
three few-shot TREC document ranking benchmarks demonstrate the consistent and
robust advantage of QDS-Transformer over previous approaches, as they either
retrofit long documents into BERT or use sparse attention without emphasizing
IR principles. We further quantify the computing complexity and demonstrates
that our sparse attention with TVM implementation is twice more efficient than
the fully-connected self-attention. All source codes, trained model, and
predictions of this work are available at
https://github.com/hallogameboy/QDS-Transformer.
