  Words change their meaning over time as well as in different contexts. The
sense-aware contextualised word embeddings (SCWEs) such as the ones produced by
XL-LEXEME by fine-tuning masked langauge models (MLMs) on Word-in-Context (WiC)
data attempt to encode such semantic changes of words within the contextualised
word embedding (CWE) spaces. Despite the superior performance of SCWEs in
contextual/temporal semantic change detection (SCD) benchmarks, it remains
unclear as to how the meaning changes are encoded in the embedding space. To
study this, we compare pre-trained CWEs and their fine-tuned versions on
contextual and temporal semantic change benchmarks under Principal Component
Analysis (PCA) and Independent Component Analysis (ICA) transformations. Our
experimental results reveal several novel insights such as (a) although there
exist a smaller number of axes that are responsible for semantic changes of
words in the pre-trained CWE space, this information gets distributed across
all dimensions when fine-tuned, and (b) in contrast to prior work studying the
geometry of CWEs, we find that PCA to better represent semantic changes than
ICA. Source code is available at https://github.com/LivNLP/svp-dims .
