  Deep neural networks have emerged as the workhorse for a large section of
robotics and control applications, especially as models for dynamical systems.
Such data-driven models are in turn used for designing and verifying autonomous
systems. They are particularly useful in modeling medical systems where data
can be leveraged to individualize treatment. In safety-critical applications,
it is important that the data-driven model is conformant to established
knowledge from the natural sciences. Such knowledge is often available or can
often be distilled into a (possibly black-box) model. For instance, an F1
racing car should conform to Newton's laws (which are encoded within a unicycle
model). In this light, we consider the following problem - given a model $M$
and a state transition dataset, we wish to best approximate the system model
while being a bounded distance away from $M$. We propose a method to guarantee
this conformance. Our first step is to distill the dataset into a few
representative samples called memories, using the idea of a growing neural gas.
Next, using these memories we partition the state space into disjoint subsets
and compute bounds that should be respected by the neural network in each
subset. This serves as a symbolic wrapper for guaranteed conformance. We argue
theoretically that this only leads to a bounded increase in approximation
error; which can be controlled by increasing the number of memories. We
experimentally show that on three case studies (Car Model, Drones, and
Artificial Pancreas), our constrained neurosymbolic models conform to specified
models (each encoding various constraints) with order-of-magnitude improvements
compared to the augmented Lagrangian and vanilla training methods. Our code can
be found at: https://github.com/kaustubhsridhar/Constrained_Models
