  Sparse linear models are one of several core tools for interpretable machine
learning, a field of emerging importance as predictive models permeate
decision-making in many domains. Unfortunately, sparse linear models are far
less flexible as functions of their input features than black-box models like
deep neural networks. With this capability gap in mind, we study a not-uncommon
situation where the input features dichotomize into two groups: explanatory
features, which are candidates for inclusion as variables in an interpretable
model, and contextual features, which select from the candidate variables and
determine their effects. This dichotomy leads us to the contextual lasso, a new
statistical estimator that fits a sparse linear model to the explanatory
features such that the sparsity pattern and coefficients vary as a function of
the contextual features. The fitting process learns this function
nonparametrically via a deep neural network. To attain sparse coefficients, we
train the network with a novel lasso regularizer in the form of a projection
layer that maps the network's output onto the space of $\ell_1$-constrained
linear models. An extensive suite of experiments on real and synthetic data
suggests that the learned models, which remain highly transparent, can be
sparser than the regular lasso without sacrificing the predictive power of a
standard deep neural network.
