  With the growing access to administrative health databases, retrospective
studies have become crucial evidence for medical treatments. Yet,
non-randomized studies frequently face selection biases, requiring mitigation
strategies. Propensity score matching (PSM) addresses these biases by selecting
comparable populations, allowing for analysis without further methodological
constraints. However, PSM has several drawbacks. Different matching methods can
produce significantly different Average Treatment Effects (ATE) for the same
task, even when meeting all validation criteria. To prevent cherry-picking the
best method, public authorities must involve field experts and engage in
extensive discussions with researchers.
  To address this issue, we introduce a novel metric, A2A, to reduce the number
of valid matches. A2A constructs artificial matching tasks that mirror the
original ones but with known outcomes, assessing each matching method's
performance comprehensively from propensity estimation to ATE estimation. When
combined with Standardized Mean Difference, A2A enhances the precision of model
selection, resulting in a reduction of up to 50% in ATE estimation errors
across synthetic tasks and up to 90% in predicted ATE variability across both
synthetic and real-world datasets. To our knowledge, A2A is the first metric
capable of evaluating outcome correction accuracy using covariates not involved
in selection.
  Computing A2A requires solving hundreds of PSMs, we therefore automate all
manual steps of the PSM pipeline. We integrate PSM methods from Python and R,
our automated pipeline, a new metric, and reproducible experiments into
popmatch, our new Python package, to enhance reproducibility and accessibility
to bias correction methods.
