  Standard NLP tasks do not incorporate several common real-world scenarios
such as seeking clarifications about the question, taking advantage of clues,
abstaining in order to avoid incorrect answers, etc. This difference in task
formulation hinders the adoption of NLP systems in real-world settings. In this
work, we take a step towards bridging this gap and present a multi-stage task
that simulates a typical human-human questioner-responder interaction such as
an interview. Specifically, the system is provided with question
simplifications, knowledge statements, examples, etc. at various stages to
improve its prediction when it is not sufficiently confident. We instantiate
the proposed task in Natural Language Inference setting where a system is
evaluated on both in-domain and out-of-domain (OOD) inputs. We conduct
comprehensive experiments and find that the multi-stage formulation of our task
leads to OOD generalization performance improvement up to 2.29% in Stage 1,
1.91% in Stage 2, 54.88% in Stage 3, and 72.02% in Stage 4 over the standard
unguided prediction. However, our task leaves a significant challenge for NLP
researchers to further improve OOD performance at each stage.
