  The need for AI systems to provide explanations for their behaviour is now
widely recognised as key to their adoption. In this paper, we examine the
problem of trustworthy AI and explore what delivering this means in practice,
with a focus on healthcare applications. Work in this area typically treats
trustworthy AI as a problem of Human-Computer Interaction involving the
individual user and an AI system. However, we argue here that this overlooks
the important part played by organisational accountability in how people reason
about and trust AI in socio-technical settings. To illustrate the importance of
organisational accountability, we present findings from ethnographic studies of
breast cancer screening and cancer treatment planning in multidisciplinary team
meetings to show how participants made themselves accountable both to each
other and to the organisations of which they are members. We use these findings
to enrich existing understandings of the requirements for trustworthy AI and to
outline some candidate solutions to the problems of making AI accountable both
to individual users and organisationally. We conclude by outlining the
implications of this for future work on the development of trustworthy AI,
including ways in which our proposed solutions may be re-used in different
application settings.
