  With the development of Large Language Models (LLMs), social biases in the
LLMs have become a crucial issue. While various benchmarks for social biases
have been provided across languages, the extent to which Japanese LLMs exhibit
social biases has not been fully investigated. In this study, we construct the
Japanese Bias Benchmark dataset for Question Answering (JBBQ) based on the
English bias benchmark BBQ, and analyze social biases in Japanese LLMs. The
results show that while current Japanese LLMs improve their accuracies on JBBQ
by instruction-tuning, their bias scores become larger. In addition, augmenting
their prompts with warning about social biases reduces the effect of biases in
some models.
