  Despite remarkable progress made in natural language processing, even the
state-of-the-art models often make incorrect predictions. Such predictions
hamper the reliability of systems and limit their widespread adoption in
real-world applications. 'Selective prediction' partly addresses the above
concern by enabling models to abstain from answering when their predictions are
likely to be incorrect. While selective prediction is advantageous, it leaves
us with a pertinent question 'what to do after abstention'. To this end, we
present an explorative study on 'Post-Abstention', a task that allows
re-attempting the abstained instances with the aim of increasing 'coverage' of
the system without significantly sacrificing its 'accuracy'. We first provide
mathematical formulation of this task and then explore several methods to solve
it. Comprehensive experiments on 11 QA datasets show that these methods lead to
considerable risk improvements -- performance metric of the Post-Abstention
task -- both in the in-domain and the out-of-domain settings. We also conduct a
thorough analysis of these results which further leads to several interesting
findings. Finally, we believe that our work will encourage and facilitate
further research in this important area of addressing the reliability of NLP
systems.
