  Recent advances to combine structured regression models and deep neural
networks for better interpretability, more expressiveness, and statistically
valid uncertainty quantification demonstrate the versatility of semi-structured
neural networks (SSNs). We show that techniques to properly identify the
contributions of the different model components in SSNs, however, lead to
suboptimal network estimation, slower convergence, and degenerated or erroneous
predictions. In order to solve these problems while preserving favorable model
properties, we propose a non-invasive post-hoc orthogonalization (PHO) that
guarantees identifiability of model components and provides better estimation
and prediction quality. Our theoretical findings are supported by numerical
experiments, a benchmark comparison as well as a real-world application to
COVID-19 infections.
