  When people receive advice while making difficult decisions, they often make
better decisions in the moment and also increase their knowledge in the
process. However, such incidental learning can only occur when people
cognitively engage with the information they receive and process this
information thoughtfully. How do people process the information and advice they
receive from AI, and do they engage with it deeply enough to enable learning?
To answer these questions, we conducted three experiments in which individuals
were asked to make nutritional decisions and received simulated AI
recommendations and explanations. In the first experiment, we found that when
people were presented with both a recommendation and an explanation before
making their choice, they made better decisions than they did when they
received no such help, but they did not learn. In the second experiment,
participants first made their own choice, and only then saw a recommendation
and an explanation from AI; this condition also resulted in improved decisions,
but no learning. However, in our third experiment, participants were presented
with just an AI explanation but no recommendation and had to arrive at their
own decision. This condition led to both more accurate decisions and learning
gains. We hypothesize that learning gains in this condition were due to deeper
engagement with explanations needed to arrive at the decisions. This work
provides some of the most direct evidence to date that it may not be sufficient
to include explanations together with AI-generated recommendation to ensure
that people engage carefully with the AI-provided information. This work also
presents one technique that enables incidental learning and, by implication,
can help people process AI recommendations and explanations more carefully.
