  A hybrid model involves the cooperation of an interpretable model and a
complex black box. At inference, any input of the hybrid model is assigned to
either its interpretable or complex component based on a gating mechanism. The
advantages of such models over classical ones are two-fold: 1) They grant users
precise control over the level of transparency of the system and 2) They can
potentially perform better than a standalone black box since redirecting some
of the inputs to an interpretable model implicitly acts as regularization.
Still, despite their high potential, hybrid models remain under-studied in the
interpretability/explainability literature. In this paper, we remedy this fact
by presenting a thorough investigation of such models from three perspectives:
Theory, Taxonomy, and Methods. First, we explore the theory behind the
generalization of hybrid models from the Probably-Approximately-Correct (PAC)
perspective. A consequence of our PAC guarantee is the existence of a sweet
spot for the optimal transparency of the system. When such a sweet spot is
attained, a hybrid model can potentially perform better than a standalone black
box. Secondly, we provide a general taxonomy for the different ways of training
hybrid models: the Post-Black-Box and Pre-Black-Box paradigms. These approaches
differ in the order in which the interpretable and complex components are
trained. We show where the state-of-the-art hybrid models Hybrid-Rule-Set and
Companion-Rule-List fall in this taxonomy. Thirdly, we implement the two
paradigms in a single method: HybridCORELS, which extends the CORELS algorithm
to hybrid modeling. By leveraging CORELS, HybridCORELS provides a certificate
of optimality of its interpretable component and precise control over
transparency. We finally show empirically that HybridCORELS is competitive with
existing hybrid models, and performs just as well as a standalone black box (or
even better) while being partly transparent.
