  With promising yet saturated results in high-resource settings, low-resource
datasets have gradually become popular benchmarks for evaluating the learning
ability of advanced neural networks (e.g., BigBench, superGLUE). Some models
even surpass humans according to benchmark test results. However, we find that
there exists a set of hard examples in low-resource settings that challenge
neural networks but are not well evaluated, which causes over-estimated
performance. We first give a theoretical analysis on which factors bring the
difficulty of low-resource learning. It then motivate us to propose a
challenging benchmark hardBench to better evaluate the learning ability, which
covers 11 datasets, including 3 computer vision (CV) datasets and 8 natural
language process (NLP) datasets. Experiments on a wide range of models show
that neural networks, even pre-trained language models, have sharp performance
drops on our benchmark, demonstrating the effectiveness on evaluating the
weaknesses of neural networks. On NLP tasks, we surprisingly find that despite
better results on traditional low-resource benchmarks, pre-trained networks,
does not show performance improvements on our benchmarks. These results
demonstrate that there are still a large robustness gap between existing models
and human-level performance.
