  Regression problems with time-series predictors are common in banking and
many other areas of application. In this paper, we use multi-head attention
networks to develop interpretable features and use them to achieve good
predictive performance. The customized attention layer explicitly uses
multiplicative interactions and builds feature-engineering heads that capture
temporal dynamics in a parsimonious manner. Convolutional layers are used to
combine multivariate time series. We also discuss methods for handling static
covariates in the modeling process. Visualization and explanation tools are
used to interpret the results and explain the relationship between the inputs
and the extracted features. Both simulation and real dataset are used to
illustrate the usefulness of the methodology. Keyword: Attention heads, Deep
neural networks, Interpretable feature engineering
