  Most existing interpretable methods explain a black-box model in a post-hoc
manner, which uses simpler models or data analysis techniques to interpret the
predictions after the model is learned. However, they (a) may derive
contradictory explanations on the same predictions given different methods and
data samples, and (b) focus on using simpler models to provide higher
descriptive accuracy at the sacrifice of prediction accuracy. To address these
issues, we propose a hybrid interpretable model that combines a piecewise
linear component and a nonlinear component. The first component describes the
explicit feature contributions by piecewise linear approximation to increase
the expressiveness of the model. The other component uses a multi-layer
perceptron to capture feature interactions and implicit nonlinearity, and
increase the prediction performance. Different from the post-hoc approaches,
the interpretability is obtained once the model is learned in the form of
feature shapes. We also provide a variant to explore higher-order interactions
among features to demonstrate that the proposed model is flexible for
adaptation. Experiments demonstrate that the proposed model can achieve good
interpretability by describing feature shapes while maintaining
state-of-the-art accuracy.
