  Supervised deep learning is most commonly applied to difficult problems
defined on large and often extensively curated datasets. Here we demonstrate
the ability of deep representation learning to address problems of
classification and regression from small and poorly formed tabular datasets by
encoding input information as abstracted sequences composed of a fixed number
of characters per input field. We find that small models have sufficient
capacity for approximation of various functions and achieve record
classification benchmark accuracy. Such models are shown to form useful
embeddings of various input features in their hidden layers, even if the
learned task does not explicitly require knowledge of those features. These
models are also amenable to input attribution, allowing for an estimation of
the importance of each input element to the model output as well as of which
inputs features are effectively embedded in the model. We present a
proof-of-concept for the application of small language models to mixed tabular
data without explicit feature engineering, cleaning, or preprocessing, relying
on the model to perform these tasks as part of the representation learning
process.
