  We introduce Holmes, a benchmark to assess the linguistic competence of
language models (LMs) - their ability to grasp linguistic phenomena. Unlike
prior prompting-based evaluations, Holmes assesses the linguistic competence of
LMs via their internal representations using classifier-based probing. In doing
so, we disentangle specific phenomena (e.g., part-of-speech of words) from
other cognitive abilities, like following textual instructions, and meet recent
calls to assess LMs' linguistic competence in isolation. Composing Holmes, we
review over 250 probing studies and feature more than 200 datasets to assess
syntax, morphology, semantics, reasoning, and discourse phenomena. Analyzing
over 50 LMs reveals that, aligned with known trends, their linguistic
competence correlates with model size. However, surprisingly, model
architecture and instruction tuning also significantly influence performance,
particularly in morphology and syntax. Finally, we propose FlashHolmes, a
streamlined version of Holmes designed to lower the high computation load while
maintaining high-ranking precision.
