  We refer to the setting where the (partial) derivatives of a neural network's
(NN's) predictions with respect to its inputs are used as additional training
signal as a derivative-constrained (DC) NN. This situation is common in
physics-informed settings in the natural sciences. We propose an integrated
RELU (IReLU) activation function to improve training of DC NNs. We also
investigate denormalization and label rescaling to help stabilize DC training.
We evaluate our methods on physics-informed settings including quantum
chemistry and Scientific Machine Learning (SciML) tasks. We demonstrate that
existing architectures with IReLU activations combined with denormalization and
label rescaling better incorporate training signal provided by derivative
constraints.
