  Deep neural networks (DNNs) are becoming prevalent, often outperforming
manually-created systems. Unfortunately, DNN models are opaque to humans, and
may behave in unexpected ways when deployed. One approach for allowing safer
deployment of DNN models calls for augmenting them with hand-crafted override
rules, which serve to override decisions made by the DNN model when certain
criteria are met. Here, we propose to bring together DNNs and the well-studied
scenario-based modeling paradigm, by expressing these override rules as simple
and intuitive scenarios. This approach can lead to override rules that are
comprehensible to humans, but are also sufficiently expressive and powerful to
increase the overall safety of the model. We describe how to extend and apply
scenario-based modeling to this new setting, and demonstrate our proposed
technique on multiple DNN models.
