  In this paper, we study mid-cap companies, i.e. publicly traded companies
with less than US $10 billion in market capitalisation. Using a large dataset
of US mid-cap companies observed over 30 years, we look to predict the default
probability term structure over the medium term and understand which data
sources (i.e. fundamental, market or pricing data) contribute most to the
default risk. Whereas existing methods typically require that data from
different time periods are first aggregated and turned into cross-sectional
features, we frame the problem as a multi-label time-series classification
problem. We adapt transformer models, a state-of-the-art deep learning model
emanating from the natural language processing domain, to the credit risk
modelling setting. We also interpret the predictions of these models using
attention heat maps. To optimise the model further, we present a custom loss
function for multi-label classification and a novel multi-channel architecture
with differential training that gives the model the ability to use all input
data efficiently. Our results show the proposed deep learning architecture's
superior performance, resulting in a 13% improvement in AUC (Area Under the
receiver operating characteristic Curve) over traditional models. We also
demonstrate how to produce an importance ranking for the different data sources
and the temporal relationships using a Shapley approach specific to these
models.
