  Large Language Models (LLMs) have been observed to perform well on a wide
range of downstream tasks when fine-tuned on domain-specific data. However,
such data may not be readily available in many applications, motivating
zero-shot or few-shot approaches using domain-adjacent models. While several
fine-tuned models for various tasks are available, finding an appropriate
domain-adjacent model for a given task is often not straight forward. In this
paper, we study DAFT-E, a framework that utilizes an Ensemble of
Domain-Adjacent Fine-Tuned Foundation Models for few-shot problems. We show
that for zero-shot problems, this ensembling method provides an accuracy
performance close to that of the single best model. With few-shot problems,
this performance improves further, at which point DEFT-E can outperform any
single domain-adjacent model while requiring much less data for domain-specific
fine-tuning.
