  Deep learning models have been the subject of study from various
perspectives, for example, their training process, interpretation,
generalization error, robustness to adversarial attacks, etc. A trained model
is defined by its decision boundaries, and therefore, many of the studies about
deep learning models speculate about the decision boundaries, and sometimes
make simplifying assumptions about them. So far, finding exact points on the
decision boundaries of trained deep models has been considered an intractable
problem. Here, we compute exact points on the decision boundaries of these
models and provide mathematical tools to investigate the surfaces that define
the decision boundaries. Through numerical results, we confirm that some of the
speculations about the decision boundaries are accurate, some of the
computational methods can be improved, and some of the simplifying assumptions
may be unreliable, for models with nonlinear activation functions. We advocate
for verification of simplifying assumptions and approximation methods, wherever
they are used. Finally, we demonstrate that the computational practices used
for finding adversarial examples can be improved and computing the closest
point on the decision boundary reveals the weakest vulnerability of a model
against adversarial attack.
