  The advent of large language models (LLMs) has enabled significant
performance gains in the field of natural language processing. However, recent
studies have found that LLMs often resort to shortcuts when performing tasks,
creating an illusion of enhanced performance while lacking generalizability in
their decision rules. This phenomenon introduces challenges in accurately
assessing natural language understanding in LLMs. Our paper provides a concise
survey of relevant research in this area and puts forth a perspective on the
implications of shortcut learning in the evaluation of language models,
specifically for NLU tasks. This paper urges more research efforts to be put
towards deepening our comprehension of shortcut learning, contributing to the
development of more robust language models, and raising the standards of NLU
evaluation in real-world scenarios.
