  We discovered secular trend bias in a drug effectiveness study for a recently
approved drug. We compared treatment outcomes between patients who received the
newly approved drug and patients exposed to the standard treatment. All
patients diagnosed after the new drug's approval date were considered. We built
a machine learning causal inference model to determine patient subpopulations
likely to respond better to the newly approved drug. After identifying the
presence of secular trend bias in our data, we attempted to adjust for the bias
in two different ways. First, we matched patients on the number of days from
the new drug's approval date that the patient's treatment (new or standard)
began. Second, we included a covariate in the model for the number of days
between the date of approval of the new drug and the treatment (new or
standard) start date. Neither approach completely mitigated the bias. Residual
bias we attribute to differences in patient disease severity or other
unmeasured patient characteristics. Had we not identified the secular trend
bias in our data, the causal inference model would have been interpreted
without consideration for this underlying bias. Being aware of, testing for,
and handling potential bias in the data is essential to diminish the
uncertainty in AI modeling.
