  The Natural Language Processing(NLP) community has been using crowd sourcing
techniques to create benchmark datasets such as General Language Understanding
and Evaluation(GLUE) for training modern Language Models such as BERT. GLUE
tasks measure the reliability scores using inter annotator metrics i.e. Cohens
Kappa. However, the reliability aspect of LMs has often been overlooked. To
counter this problem, we explore a knowledge-guided LM ensembling approach that
leverages reinforcement learning to integrate knowledge from ConceptNet and
Wikipedia as knowledge graph embeddings. This approach mimics human annotators
resorting to external knowledge to compensate for information deficits in the
datasets. Across nine GLUE datasets, our research shows that ensembling
strengthens reliability and accuracy scores, outperforming state of the art.
