  There is an emerging trend in the reinforcement learning for healthcare
literature. In order to prepare longitudinal, irregularly sampled, clinical
datasets for reinforcement learning algorithms, many researchers will resample
the time series data to short, regular intervals and use
last-observation-carried-forward (LOCF) imputation to fill in these gaps.
Typically, they will not maintain any explicit information about which values
were imputed. In this work, we (1) call attention to this practice and discuss
its potential implications; (2) propose an alternative representation of the
patient state that addresses some of these issues; and (3) demonstrate in a
novel but representative clinical dataset that our alternative representation
yields consistently better results for achieving optimal control, as measured
by off-policy policy evaluation, compared to representations that do not
incorporate missingness information.
