  Given the success of Large Language Models (LLMs), there has been
considerable interest in studying the properties of model activations. The
literature overwhelmingly agrees that LLM representations are dominated by a
few "outlier dimensions" with exceedingly high variance and magnitude. Several
studies in Natural Language Processing (NLP) have sought to mitigate the impact
of such outlier dimensions and force LLMs to be isotropic (i.e., have uniform
variance across all dimensions in embedding space). Isotropy is thought to be a
desirable property for LLMs that improves model performance and more closely
aligns textual representations with human intuition. However, many of the
claims regarding isotropy in NLP have been based on the average cosine
similarity of embeddings, which has recently been shown to be a flawed measure
of isotropy. In this paper, we propose I-STAR: IsoScore*-based STable
Anisotropic Regularization, a novel regularization method that can be used to
increase or decrease levels of isotropy in embedding space during training.
I-STAR uses IsoScore*, the first accurate measure of isotropy that is both
differentiable and stable on mini-batch computations. In contrast to several
previous works, we find that decreasing isotropy in contextualized embeddings
improves performance on the majority of tasks and models considered in this
paper.
