  Post-approval safety surveillance of medical products using observational
healthcare data can help identify safety issues beyond those found in
pre-approval trials. When testing sequentially as data accrue, maximum
sequential probability ratio testing (MaxSPRT) is a common approach to
maintaining nominal type 1 error. However, the true type 1 error may still
deviate from the specified one because of systematic error due to the
observational nature of the analysis. This systematic error may persist even
after controlling for known confounders. Here we propose to address this issue
by combing MaxSPRT with empirical calibration. In empirical calibration, we
assume uncertainty about the systematic error in our analysis, the source of
uncertainty commonly overlooked in practice. We infer a probability
distribution of systematic error by relying on a large set of negative
controls: exposure-outcome where no causal effect is believed to exist.
Integrating this distribution into our test statistics has previously been
shown to restore type 1 error to nominal. Here we show how we can calibrate the
critical value central to MaxSPRT. We evaluate this novel approach using
simulations and real electronic health records, using H1N1 vaccinations during
the 2009-2010 season as an example. Results show that combining empirical
calibration with MaxSPRT restores nominal type 1 error. In our real-world
example, adjusting for systematic error using empirical calibration has a
larger impact than, and hence is just as essential as, adjusting for sequential
testing using MaxSPRT. We recommend performing both, using the method described
here.
