  Randomized trials are considered the gold standard for making informed
decisions in medicine, yet they often lack generalizability to the patient
populations in clinical practice. Observational studies, on the other hand,
cover a broader patient population but are prone to various biases. Thus,
before using an observational study for decision-making, it is crucial to
benchmark its treatment effect estimates against those derived from a
randomized trial. We propose a novel strategy to benchmark observational
studies beyond the average treatment effect. First, we design a statistical
test for the null hypothesis that the treatment effects estimated from the two
studies, conditioned on a set of relevant features, differ up to some
tolerance. We then estimate an asymptotically valid lower bound on the maximum
bias strength for any subgroup in the observational study. Finally, we validate
our benchmarking strategy in a real-world setting and show that it leads to
conclusions that align with established medical knowledge.
