  Eliciting feedback from end users of NLP models can be beneficial for
improving models. However, how should we present model responses to users so
they are most amenable to be corrected from user feedback? Further, what
properties do users value to understand and trust responses? We answer these
questions by analyzing the effect of rationales (or explanations) generated by
QA models to support their answers. We specifically consider decomposed QA
models that first extract an intermediate rationale based on a context and a
question and then use solely this rationale to answer the question. A rationale
outlines the approach followed by the model to answer the question. Our work
considers various formats of these rationales that vary according to
well-defined properties of interest. We sample rationales from language models
using few-shot prompting for two datasets, and then perform two user studies.
First, we present users with incorrect answers and corresponding rationales in
various formats and ask them to provide natural language feedback to revise the
rationale. We then measure the effectiveness of this feedback in patching these
rationales through in-context learning. The second study evaluates how well
different rationale formats enable users to understand and trust model answers,
when they are correct. We find that rationale formats significantly affect how
easy it is (1) for users to give feedback for rationales, and (2) for models to
subsequently execute this feedback. In addition, formats with attributions to
the context and in-depth reasoning significantly enhance user-reported
understanding and trust of model outputs.
