  Natural language understanding (NLU) tasks face a non-trivial amount of
ambiguous samples where veracity of their labels is debatable among annotators.
NLU models should thus account for such ambiguity, but they approximate the
human opinion distributions quite poorly and tend to produce over-confident
predictions. To address this problem, we must consider how to exactly capture
the degree of relationship between each sample and its candidate classes. In
this work, we propose a novel method with deep model compression and show how
such relationship can be accounted for. We see that more reasonably represented
relationships can be discovered in the lower layers and that validation
accuracies are converging at these layers, which naturally leads to layer
pruning. We also see that distilling the relationship knowledge from a lower
layer helps models produce better distribution. Experimental results
demonstrate that our method makes substantial improvement on quantifying
ambiguity without gold distribution labels. As positive side-effects, our
method is found to reduce the model size significantly and improve latency,
both attractive aspects of NLU products.
