  We introduce bipol, a new metric with explainability, for estimating social
bias in text data. Harmful bias is prevalent in many online sources of data
that are used for training machine learning (ML) models. In a step to address
this challenge we create a novel metric that involves a two-step process:
corpus-level evaluation based on model classification and sentence-level
evaluation based on (sensitive) term frequency (TF). After creating new models
to detect bias along multiple axes using SotA architectures, we evaluate two
popular NLP datasets (COPA and SQUAD). As additional contribution, we created a
large dataset (with almost 2 million labelled samples) for training models in
bias detection and make it publicly available. We also make public our codes.
