  Open Domain Question Answering (ODQA) on a large-scale corpus of documents
(e.g. Wikipedia) is a key challenge in computer science. Although
transformer-based language models such as Bert have shown on SQuAD the ability
to surpass humans for extracting answers in small passages of text, they suffer
from their high complexity when faced to a much larger search space. The most
common way to tackle this problem is to add a preliminary Information Retrieval
step to heavily filter the corpus and only keep the relevant passages. In this
paper, we propose a more direct and complementary solution which consists in
applying a generic change in the architecture of transformer-based models to
delay the attention between subparts of the input and allow a more efficient
management of computations. The resulting variants are competitive with the
original models on the extractive task and allow, on the ODQA setting, a
significant speedup and even a performance improvement in many cases.
