  Masked language models (MLMs) have contributed to drastic performance
improvements with regard to zero anaphora resolution (ZAR). To further improve
this approach, in this study, we made two proposals. The first is a new
pretraining task that trains MLMs on anaphoric relations with explicit
supervision, and the second proposal is a new finetuning method that remedies a
notorious issue, the pretrain-finetune discrepancy. Our experiments on Japanese
ZAR demonstrated that our two proposals boost the state-of-the-art performance,
and our detailed analysis provides new insights on the remaining challenges.
