  Rule-based language processing systems have been overshadowed by neural
systems in terms of utility, but it remains unclear whether neural NLP systems,
in practice, learn the grammar rules that humans use. This work aims to shed
light on the issue by evaluating state-of-the-art LLMs in a task of
morphological analysis of complex Finnish noun forms. We generate the forms
using an FST tool, and they are unlikely to have occurred in the training sets
of the LLMs, therefore requiring morphological generalisation capacity. We find
that GPT-4-turbo has some difficulties in the task while GPT-3.5-turbo
struggles and smaller models Llama2-70B and Poro-34B fail nearly completely.
