  Deep neural networks (DNNs) have become a crucial instrument in the software
development toolkit, due to their ability to efficiently solve complex
problems. Nevertheless, DNNs are highly opaque, and can behave in an unexpected
manner when they encounter unfamiliar input. One promising approach for
addressing this challenge is by extending DNN-based systems with hand-crafted
override rules, which override the DNN's output when certain conditions are
met. Here, we advocate crafting such override rules using the well-studied
scenario-based modeling paradigm, which produces rules that are simple,
extensible, and powerful enough to ensure the safety of the DNN, while also
rendering the system more translucent. We report on two extensive case studies,
which demonstrate the feasibility of the approach; and through them, propose an
extension to scenario-based modeling, which facilitates its integration with
DNN components. We regard this work as a step towards creating safer and more
reliable DNN-based systems and models.
