  In recent years, word embeddings have been widely used to measure biases in
texts. Even if they have proven to be effective in detecting a wide variety of
biases, metrics based on word embeddings lack transparency and
interpretability. We analyze an alternative PMI-based metric to quantify biases
in texts. It can be expressed as a function of conditional probabilities, which
provides a simple interpretation in terms of word co-occurrences. We also prove
that it can be approximated by an odds ratio, which allows estimating
confidence intervals and statistical significance of textual biases. This
approach produces similar results to metrics based on word embeddings when
capturing gender gaps of the real world embedded in large corpora.
