  Recent improvements in large language models (LLMs) have significantly
enhanced natural language processing (NLP) applications. However, these models
can also inherit and perpetuate biases from their training data. Addressing
this issue is crucial, yet many existing datasets do not offer evaluation
across diverse NLP tasks. To tackle this, we introduce the Bias Evaluations
Across Domains (BEADs) dataset, designed to support a wide range of NLP tasks,
including text classification, bias entity recognition, bias quantification,
and benign language generation. BEADs uses AI-driven annotation combined with
experts' verification to provide reliable labels. This method overcomes the
limitations of existing datasets that typically depend on crowd-sourcing,
expert-only annotations with limited bias evaluations, or unverified AI
labeling. Our empirical analysis shows that BEADs is effective in detecting and
reducing biases across different language models, with smaller models
fine-tuned on BEADs often outperforming LLMs in bias classification tasks.
However, these models may still exhibit biases towards certain demographics.
Fine-tuning LLMs with our benign language data also reduces biases while
preserving the models' knowledge. Our findings highlight the importance of
comprehensive bias evaluation and the potential of targeted fine-tuning for
reducing the bias of LLMs. We are making BEADs publicly available at
https://huggingface.co/datasets/shainar/BEAD
  Warning: This paper contains examples that may be considered offensive.
