  As the applications of Natural Language Processing (NLP) in sensitive areas
like Political Profiling, Review of Essays in Education, etc. proliferate,
there is a great need for increasing transparency in NLP models to build trust
with stakeholders and identify biases. A lot of work in Explainable AI has
aimed to devise explanation methods that give humans insights into the workings
and predictions of NLP models. While these methods distill predictions from
complex models like Neural Networks into consumable explanations, how humans
understand these explanations is still widely unexplored. Innate human
tendencies and biases can handicap the understanding of these explanations in
humans, and can also lead to them misjudging models and predictions as a
result. We designed a randomized survey-based experiment to understand the
effectiveness of saliency-based Post-hoc explainability methods in Natural
Language Processing. The result of the experiment showed that humans have a
tendency to accept explanations with a less critical view.
