  We propose Differentiable Window, a new neural module and general purpose
component for dynamic window selection. While universally applicable, we
demonstrate a compelling use case of utilizing Differentiable Window to improve
standard attention modules by enabling more focused attentions over the input
regions. We propose two variants of Differentiable Window, and integrate them
within the Transformer architecture in two novel ways. We evaluate our proposed
approach on a myriad of NLP tasks, including machine translation, sentiment
analysis, subject-verb agreement and language modeling. Our experimental
results demonstrate consistent and sizable improvements across all tasks.
