  The few-shot natural language understanding (NLU) task has attracted much
recent attention. However, prior methods have been evaluated under a disparate
set of protocols, which hinders fair comparison and measuring progress of the
field. To address this issue, we introduce an evaluation framework that
improves previous evaluation procedures in three key aspects, i.e., test
performance, dev-test correlation, and stability. Under this new evaluation
framework, we re-evaluate several state-of-the-art few-shot methods for NLU
tasks. Our framework reveals new insights: (1) both the absolute performance
and relative gap of the methods were not accurately estimated in prior
literature; (2) no single method dominates most tasks with consistent
performance; (3) improvements of some methods diminish with a larger pretrained
model; and (4) gains from different methods are often complementary and the
best combined model performs close to a strong fully-supervised baseline. We
open-source our toolkit, FewNLU, that implements our evaluation framework along
with a number of state-of-the-art methods.
