  Language model-based instruction-following systems have lately shown
increasing performance on many benchmark tasks, demonstrating the capability of
adapting to a broad variety of instructions. However, such systems are often
not designed to be transparent about their limitations; a user may easily
prompt a model with an instruction without any idea of whether the responses
should be expected to be accurate, or if the system is even capable of
performing the task. We propose a third party performance prediction framework,
where a separate model is trained to predict the metric resulting from
evaluating an instruction-following system on a task while assuming access only
to its inputs and outputs at inference time. We perform this analysis with a
variety of both open and closed instruction-following models as well as
multiple performance predictors, and examine the effect of various factors such
as model size, number of training tasks, and prompt format. Our findings
indicate that third-party performance prediction is very challenging, and much
work remains in developing predictors that can automatically reveal the
limitations of modern instruction-following natural language processing
systems.
