  Existing learning methods often struggle to balance interpretability and
predictive performance. While models like nearest neighbors and non-negative
matrix factorization (NMF) offer high interpretability, their predictive
performance on supervised learning tasks is often limited. In contrast, neural
networks based on the multi-layer perceptron (MLP) support the modular
construction of expressive architectures and tend to have better recognition
accuracy but are often regarded as black boxes in terms of interpretability.
Our approach aims to strike a better balance between these two aspects through
the use of a building block based on NMF that incorporates supervised neural
network training methods to achieve high predictive performance while retaining
the desirable interpretability properties of NMF. We evaluate our Predictive
Factorized Coupling (PFC) block on small datasets and show that it achieves
competitive predictive performance with MLPs while also offering improved
interpretability. We demonstrate the benefits of this approach in various
scenarios, such as continual learning, training on non-i.i.d. data, and
knowledge removal after training. Additionally, we show examples of using the
PFC block to build more expressive architectures, including a fully-connected
residual network as well as a factorized recurrent neural network (RNN) that
performs competitively with vanilla RNNs while providing improved
interpretability. The PFC block uses an iterative inference algorithm that
converges to a fixed point, making it possible to trade off accuracy vs
computation after training but also currently preventing its use as a general
MLP replacement in some scenarios such as training on very large datasets. We
provide source code at https://github.com/bkvogel/pfc
