  Structured prediction requires searching over a combinatorial number of
structures. To tackle it, we introduce SparseMAP: a new method for sparse
structured inference, and its natural loss function. SparseMAP automatically
selects only a few global structures: it is situated between MAP inference,
which picks a single structure, and marginal inference, which assigns
probability mass to all structures, including implausible ones. Importantly,
SparseMAP can be computed using only calls to a MAP oracle, making it
applicable to problems with intractable marginal inference, e.g., linear
assignment. Sparsity makes gradient backpropagation efficient regardless of the
structure, enabling us to augment deep neural networks with generic and sparse
structured hidden layers. Experiments in dependency parsing and natural
language inference reveal competitive accuracy, improved interpretability, and
the ability to capture natural language ambiguities, which is attractive for
pipeline systems.
