  The hindering problem in facial expression recognition (FER) is the presence
of inaccurate annotations referred to as noisy annotations in the datasets.
These noisy annotations are present in the datasets inherently because the
labeling is subjective to the annotator, clarity of the image, etc. Recent
works use sample selection methods to solve this noisy annotation problem in
FER. In our work, we use a dynamic adaptive threshold to separate confident
samples from non-confident ones so that our learning won't be hampered due to
non-confident samples. Instead of discarding the non-confident samples, we
impose consistency in the negative classes of those non-confident samples to
guide the model to learn better in the positive class. Since FER datasets
usually come with 7 or 8 classes, we can correctly guess a negative class by
85% probability even by choosing randomly. By learning "which class a sample
doesn't belong to", the model can learn "which class it belongs to" in a better
manner. We demonstrate proposed framework's effectiveness using quantitative as
well as qualitative results. Our method performs better than the baseline by a
margin of 4% to 28% on RAFDB and 3.3% to 31.4% on FERPlus for various levels of
synthetic noisy labels in the aforementioned datasets.
