  While fine-tuning pre-trained models for downstream classification is the
conventional paradigm in NLP, often task-specific nuances may not get captured
in the resultant models. Specifically, for tasks that take two inputs and
require the output to be invariant of the order of the inputs, inconsistency is
often observed in the predicted labels or confidence scores. We highlight this
model shortcoming and apply a consistency loss function to alleviate
inconsistency in symmetric classification. Our results show an improved
consistency in predictions for three paraphrase detection datasets without a
significant drop in the accuracy scores. We examine the classification
performance of six datasets (both symmetric and non-symmetric) to showcase the
strengths and limitations of our approach.
