  There is increasing interest in developing AIs for assisting human
decision-making in high-stakes tasks, such as medical diagnosis, for the
purpose of improving decision quality and reducing cognitive strain. Mainstream
approaches team up an expert with a machine learning model to which safer
decisions are offloaded, thus letting the former focus on cases that demand
their attention. his separation of responsibilities setup, however, is
inadequate for high-stakes scenarios. On the one hand, the expert may end up
over-relying on the machine's decisions due to anchoring bias, thus losing the
human oversight that is increasingly being required by regulatory agencies to
ensure trustworthy AI. On the other hand, the expert is left entirely
unassisted on the (typically hardest) decisions on which the model abstained.
As a remedy, we introduce learning to guide (LTG), an alternative framework in
which - rather than taking control from the human expert - the machine provides
guidance useful for decision making, and the human is entirely responsible for
coming up with a decision. In order to ensure guidance is interpretable} and
task-specific, we develop SLOG, an approach for turning any vision-language
model into a capable generator of textual guidance by leveraging a modicum of
human feedback. Our empirical evaluation highlights the promise of \method on a
challenging, real-world medical diagnosis task.
