  Discriminatory gender biases have been found in Pre-trained Language Models
(PLMs) for multiple languages. In Natural Language Inference (NLI), existing
bias evaluation methods have focused on the prediction results of one specific
label out of three labels, such as neutral. However, such evaluation methods
can be inaccurate since unique biased inferences are associated with unique
prediction labels. Addressing this limitation, we propose a bias evaluation
method for PLMs, called NLI-CoAL, which considers all the three labels of NLI
task. First, we create three evaluation data groups that represent different
types of biases. Then, we define a bias measure based on the corresponding
label output of each data group. In the experiments, we introduce a
meta-evaluation technique for NLI bias measures and use it to confirm that our
bias measure can distinguish biased, incorrect inferences from non-biased
incorrect inferences better than the baseline, resulting in a more accurate
bias evaluation. We create the datasets in English, Japanese, and Chinese, and
successfully validate the compatibility of our bias measure across multiple
languages. Lastly, we observe the bias tendencies in PLMs of different
languages. To our knowledge, we are the first to construct evaluation datasets
and measure PLMs' bias from NLI in Japanese and Chinese.
