  Deep learning models for natural language processing (NLP) are increasingly
adopted and deployed by analysts without formal training in NLP or machine
learning (ML). However, the documentation intended to convey the model's
details and appropriate use is tailored primarily to individuals with ML or NLP
expertise. To address this gap, we conduct a design inquiry into interactive
model cards, which augment traditionally static model cards with affordances
for exploring model documentation and interacting with the models themselves.
Our investigation consists of an initial conceptual study with experts in ML,
NLP, and AI Ethics, followed by a separate evaluative study with non-expert
analysts who use ML models in their work. Using a semi-structured interview
format coupled with a think-aloud protocol, we collected feedback from a total
of 30 participants who engaged with different versions of standard and
interactive model cards. Through a thematic analysis of the collected data, we
identified several conceptual dimensions that summarize the strengths and
limitations of standard and interactive model cards, including: stakeholders;
design; guidance; understandability & interpretability; sensemaking &
skepticism; and trust & safety. Our findings demonstrate the importance of
carefully considered design and interactivity for orienting and supporting
non-expert analysts using deep learning models, along with a need for
consideration of broader sociotechnical contexts and organizational dynamics.
We have also identified design elements, such as language, visual cues, and
warnings, among others, that support interactivity and make non-interactive
content accessible. We summarize our findings as design guidelines and discuss
their implications for a human-centered approach towards AI/ML documentation.
