  Multilingual evaluation benchmarks usually contain limited high-resource
languages and do not test models for specific linguistic capabilities.
CheckList is a template-based evaluation approach that tests models for
specific capabilities. The CheckList template creation process requires native
speakers, posing a challenge in scaling to hundreds of languages. In this work,
we explore multiple approaches to generate Multilingual CheckLists. We device
an algorithm - Template Extraction Algorithm (TEA) for automatically extracting
target language CheckList templates from machine translated instances of a
source language templates. We compare the TEA CheckLists with CheckLists
created with different levels of human intervention. We further introduce
metrics along the dimensions of cost, diversity, utility, and correctness to
compare the CheckLists. We thoroughly analyze different approaches to creating
CheckLists in Hindi. Furthermore, we experiment with 9 more different
languages. We find that TEA followed by human verification is ideal for scaling
Checklist-based evaluation to multiple languages while TEA gives a good
estimates of model performance.
