  Natural language understanding (NLU) has made massive progress driven by
large benchmarks, but benchmarks often leave a long tail of infrequent
phenomena underrepresented. We reflect on the question: have transfer learning
methods sufficiently addressed the poor performance of benchmark-trained models
on the long tail? We conceptualize the long tail using macro-level dimensions
(e.g., underrepresented genres, topics, etc.), and perform a qualitative
meta-analysis of 100 representative papers on transfer learning research for
NLU. Our analysis asks three questions: (i) Which long tail dimensions do
transfer learning studies target? (ii) Which properties of adaptation methods
help improve performance on the long tail? (iii) Which methodological gaps have
greatest negative impact on long tail performance? Our answers highlight major
avenues for future research in transfer learning for the long tail. Lastly,
using our meta-analysis framework, we perform a case study comparing the
performance of various adaptation methods on clinical narratives, which
provides interesting insights that may enable us to make progress along these
future avenues.
