  Alignment of the language model with human preferences is a common approach
to making a language model useful to end users. However, most alignment work is
done in English, and human preference datasets are dominated by English,
reflecting only the preferences of English-speaking annotators. Nevertheless,
it is common practice to use the English preference data, either directly or by
translating it into the target language, when aligning a multilingual language
model. The question is whether such an alignment strategy marginalizes the
preference of non-English speaking users. To this end, we investigate the
effect of aligning Japanese language models with (mostly) English resources. In
particular, we focus on evaluating whether the commonsense morality of the
resulting fine-tuned models is aligned with Japanese culture using the
JCommonsenseMorality (JCM) and ETHICS datasets. The experimental results show
that the fine-tuned model outperforms the SFT model. However, it does not
demonstrate the same level of improvement as a model fine-tuned using the JCM,
suggesting that while some aspects of commonsense morality are transferable,
others may not be.
