  Organizations including companies, nonprofits, governments, and academic
institutions are increasingly developing, deploying, and utilizing artificial
intelligence (AI) tools. Responsible AI (RAI) governance approaches at
organizations have emerged as important mechanisms to address potential AI
risks and harms. In this work, we interviewed 17 technical contributors across
organization types (Academic, Government, Industry, Nonprofit) and sectors
(Finance, Health, Tech, Other) about their experiences with internal RAI
governance. Our findings illuminated the variety of organizational definitions
of RAI and accompanying internal governance approaches. We summarized the first
detailed findings on algorithm review boards (ARBs) and similar review
committees in practice, including their membership, scope, and measures of
success. We confirmed known robust model governance in finance sectors and
revealed extensive algorithm and AI governance with ARB-like review boards in
health sectors. Our findings contradict the idea that Institutional Review
Boards alone are sufficient for algorithm governance and posit that ARBs are
among the more impactful internal RAI governance approaches. Our results
suggest that integration with existing internal regulatory approaches and
leadership buy-in are among the most important attributes for success and that
financial tensions are the greatest challenge to effective organizational RAI.
We make a variety of suggestions for how organizational partners can learn from
these findings when building their own internal RAI frameworks. We outline
future directions for developing and measuring effectiveness of ARBs and other
internal RAI governance approaches.
