  Despite their remarkable ability to capture linguistic nuances across diverse
languages, questions persist regarding the degree of alignment between
languages in multilingual embeddings. Drawing inspiration from research on
high-dimensional representations in neural language models, we employ
clustering to uncover latent concepts within multilingual models. Our analysis
focuses on quantifying the \textit{alignment} and \textit{overlap} of these
concepts across various languages within the latent space. To this end, we
introduce two metrics \CA{} and \CO{} aimed at quantifying these aspects,
enabling a deeper exploration of multilingual embeddings. Our study encompasses
three multilingual models (\texttt{mT5}, \texttt{mBERT}, and \texttt{XLM-R})
and three downstream tasks (Machine Translation, Named Entity Recognition, and
Sentiment Analysis). Key findings from our analysis include: i) deeper layers
in the network demonstrate increased cross-lingual \textit{alignment} due to
the presence of language-agnostic concepts, ii) fine-tuning of the models
enhances \textit{alignment} within the latent space, and iii) such
task-specific calibration helps in explaining the emergence of zero-shot
capabilities in the models.\footnote{The code is available at
\url{https://github.com/baselmousi/multilingual-latent-concepts}}
