  Large Language Models (LLMs) have demonstrated unprecedented prowess across
various natural language processing tasks in various application domains.
Recent studies show that LLMs can be leveraged to perform lexical semantic
tasks, such as Knowledge Base Completion (KBC) or Ontology Learning (OL).
However, it has not effectively been verified whether their success is due to
their ability to reason over unstructured or semi-structured data, or their
effective learning of linguistic patterns and senses alone. This unresolved
question is particularly crucial when dealing with domain-specific data, where
the lexical senses and their meaning can completely differ from what a LLM has
learned during its training stage. This paper investigates the following
question: Do LLMs really adapt to domains and remain consistent in the
extraction of structured knowledge, or do they only learn lexical senses
instead of reasoning? To answer this question and, we devise a controlled
experiment setup that uses WordNet to synthesize parallel corpora, with English
and gibberish terms. We examine the differences in the outputs of LLMs for each
corpus in two OL tasks: relation extraction and taxonomy discovery. Empirical
results show that, while adapting to the gibberish corpora, off-the-shelf LLMs
do not consistently reason over semantic relationships between concepts, and
instead leverage senses and their frame. However, fine-tuning improves the
performance of LLMs on lexical semantic tasks even when the domain-specific
terms are arbitrary and unseen during pre-training, hinting at the
applicability of pre-trained LLMs for OL.
