  Current computational approaches for analysing or generating code-mixed
sentences do not explicitly model "naturalness" or "acceptability" of
code-mixed sentences, but rely on training corpora to reflect distribution of
acceptable code-mixed sentences. Modelling human judgement for the
acceptability of code-mixed text can help in distinguishing natural code-mixed
text and enable quality-controlled generation of code-mixed text. To this end,
we construct Cline - a dataset containing human acceptability judgements for
English-Hindi (en-hi) code-mixed text. Cline is the largest of its kind with
16,642 sentences, consisting of samples sourced from two sources: synthetically
generated code-mixed text and samples collected from online social media. Our
analysis establishes that popular code-mixing metrics such as CMI, Number of
Switch Points, Burstines, which are used to filter/curate/compare code-mixed
corpora have low correlation with human acceptability judgements, underlining
the necessity of our dataset. Experiments using Cline demonstrate that simple
Multilayer Perceptron (MLP) models trained solely on code-mixing metrics are
outperformed by fine-tuned pre-trained Multilingual Large Language Models
(MLLMs). Specifically, XLM-Roberta and Bernice outperform IndicBERT across
different configurations in challenging data settings. Comparison with
ChatGPT's zero and fewshot capabilities shows that MLLMs fine-tuned on larger
data outperform ChatGPT, providing scope for improvement in code-mixed tasks.
Zero-shot transfer from English-Hindi to English-Telugu acceptability judgments
using our model checkpoints proves superior to random baselines, enabling
application to other code-mixed language pairs and providing further avenues of
research. We publicly release our human-annotated dataset, trained checkpoints,
code-mix corpus, and code for data generation and model training.
