  Many diverse phenomena in nature often inherently encode both short- and
long-term temporal dependencies, which especially result from the direction of
the flow of time. In this respect, we discovered experimental evidence
suggesting that interrelations of these events are higher for closer time
stamps. However, to be able for attention-based models to learn these
regularities in short-term dependencies, it requires large amounts of data,
which are often infeasible. This is because, while they are good at learning
piece-wise temporal dependencies, attention-based models lack structures that
encode biases in time series. As a resolution, we propose a simple and
efficient method that enables attention layers to better encode the short-term
temporal bias of these data sets by applying learnable, adaptive kernels
directly to the attention matrices. We chose various prediction tasks for the
experiments using Electronic Health Records (EHR) data sets since they are
great examples with underlying long- and short-term temporal dependencies. Our
experiments show exceptional classification results compared to best-performing
models on most tasks and data sets.
