  Many major works in social science employ matching to make causal
conclusions, but different matches on the same data may produce different
treatment effect estimates, even when they achieve similar balance or minimize
the same loss function. We discuss reasons and consequences of this problem. We
present evidence of this problem by replicating ten papers that use matching
and we find that different popular matching algorithms produce inconsistent
results. We introduce Matching Bounds: a finite-sample, nonstochastic method
that allows analysts to know whether a matched sample that produces different
results with the same levels of balance and overall match quality could be
obtained from their data. We apply Matching Bounds to a replication of two
studies and show that in one case results are robust to this issue and in
another they are not.
