  Machine learning (ML) is increasingly being used in image retrieval systems
for medical decision making. One application of ML is to retrieve visually
similar medical images from past patients (e.g. tissue from biopsies) to
reference when making a medical decision with a new patient. However, no
algorithm can perfectly capture an expert's ideal notion of similarity for
every case: an image that is algorithmically determined to be similar may not
be medically relevant to a doctor's specific diagnostic needs. In this paper,
we identified the needs of pathologists when searching for similar images
retrieved using a deep learning algorithm, and developed tools that empower
users to cope with the search algorithm on-the-fly, communicating what types of
similarity are most important at different moments in time. In two evaluations
with pathologists, we found that these refinement tools increased the
diagnostic utility of images found and increased user trust in the algorithm.
The tools were preferred over a traditional interface, without a loss in
diagnostic accuracy. We also observed that users adopted new strategies when
using refinement tools, re-purposing them to test and understand the underlying
algorithm and to disambiguate ML errors from their own errors. Taken together,
these findings inform future human-ML collaborative systems for expert
decision-making.
