  We investigate the effectiveness of ensembles of pretrained transformer-based
language models on short answer questions using the Kaggle Automated Short
Answer Scoring dataset. We fine-tune a collection of popular small, base, and
large pretrained transformer-based language models, and train one feature-base
model on the dataset with the aim of testing ensembles of these models. We used
an early stopping mechanism and hyperparameter optimization in training. We
observe that generally that the larger models perform slightly better, however,
they still fall short of state-of-the-art results one their own. Once we
consider ensembles of models, there are ensembles of a number of large networks
that do produce state-of-the-art results, however, these ensembles are too
large to realistically be put in a production environment.
