  We present IndoNLI, the first human-elicited NLI dataset for Indonesian. We
adapt the data collection protocol for MNLI and collect nearly 18K sentence
pairs annotated by crowd workers and experts. The expert-annotated data is used
exclusively as a test set. It is designed to provide a challenging test-bed for
Indonesian NLI by explicitly incorporating various linguistic phenomena such as
numerical reasoning, structural changes, idioms, or temporal and spatial
reasoning. Experiment results show that XLM-R outperforms other pre-trained
models in our data. The best performance on the expert-annotated data is still
far below human performance (13.4% accuracy gap), suggesting that this test set
is especially challenging. Furthermore, our analysis shows that our
expert-annotated data is more diverse and contains fewer annotation artifacts
than the crowd-annotated data. We hope this dataset can help accelerate
progress in Indonesian NLP research.
