  Questions of fairness, robustness, and transparency are paramount to address
before deploying NLP systems. Central to these concerns is the question of
reliability: Can NLP systems reliably treat different demographics fairly and
function correctly in diverse and noisy environments? To address this, we argue
for the need for reliability testing and contextualize it among existing work
on improving accountability. We show how adversarial attacks can be reframed
for this goal, via a framework for developing reliability tests. We argue that
reliability testing -- with an emphasis on interdisciplinary collaboration --
will enable rigorous and targeted testing, and aid in the enactment and
enforcement of industry standards.
