  Automated decision-making systems, especially those based on natural language
processing, are pervasive in our lives. They are not only behind the internet
search engines we use daily, but also take more critical roles: selecting
candidates for a job, determining suspects of a crime, diagnosing autism and
more. Such automated systems make errors, which may be harmful in many ways, be
it because of the severity of the consequences (as in health issues) or because
of the sheer number of people they affect. When errors made by an automated
system affect a population more than others, we call the system
\textit{biased}.
  Most modern natural language technologies are based on artifacts obtained
from enormous volumes of text using machine learning, namely language models
and word embeddings. Since they are created by applying subsymbolic machine
learning, mostly artificial neural networks, they are opaque and practically
uninterpretable by direct inspection, thus making it very difficult to audit
them.
  In this paper, we present a methodology that spells out how social
scientists, domain experts, and machine learning experts can collaboratively
explore biases and harmful stereotypes in word embeddings and large language
models. Our methodology is based on the following principles:
  * focus on the linguistic manifestations of discrimination on word embeddings
and language models, not on the mathematical properties of the models * reduce
the technical barrier for discrimination experts%, be it social scientists,
domain experts or other * characterize through a qualitative exploratory
process in addition to a metric-based approach * address mitigation as part of
the training process, not as an afterthought
