  The Multisource AI Scorecard Table (MAST) is a checklist tool based on
analytic tradecraft standards to inform the design and evaluation of
trustworthy AI systems. In this study, we evaluate whether MAST is associated
with people's trust perceptions in AI-enabled decision support systems
(AI-DSSs). Evaluating trust in AI-DSSs poses challenges to researchers and
practitioners. These challenges include identifying the components,
capabilities, and potential of these systems, many of which are based on the
complex deep learning algorithms that drive DSS performance and preclude
complete manual inspection. We developed two interactive, AI-DSS test
environments using the MAST criteria. One emulated an identity verification
task in security screening, and another emulated a text summarization system to
aid in an investigative reporting task. Each test environment had one version
designed to match low-MAST ratings, and another designed to match high-MAST
ratings, with the hypothesis that MAST ratings would be positively related to
the trust ratings of these systems. A total of 177 subject matter experts were
recruited to interact with and evaluate these systems. Results generally show
higher MAST ratings for the high-MAST conditions compared to the low-MAST
groups, and that measures of trust perception are highly correlated with the
MAST ratings. We conclude that MAST can be a useful tool for designing and
evaluating systems that will engender high trust perceptions, including AI-DSS
that may be used to support visual screening and text summarization tasks.
However, higher MAST ratings may not translate to higher joint performance.
