  Deep Neural Networks (DNNs) are widely used for their ability to effectively
approximate large classes of functions. This flexibility, however, makes the
strict enforcement of constraints on DNNs an open problem. Here we present a
framework that, under mild assumptions, allows the exact enforcement of
constraints on parameterized sets of functions such as DNNs. Instead of
imposing "soft'' constraints via additional terms in the loss, we restrict (a
subset of) the DNN parameters to a submanifold on which the constraints are
satisfied exactly throughout the entire training procedure. We focus on
constraints that are outside the scope of equivariant networks used in
Geometric Deep Learning. As a major example of the framework, we restrict
filters of a Convolutional Neural Network (CNN) to be wavelets, and apply these
wavelet networks to the task of contour prediction in the medical domain.
