  Offline reinforcement learning has shown promise for solving tasks in
safety-critical settings, such as clinical decision support. Its application,
however, has been limited by the lack of interpretability and interactivity for
clinicians. To address these challenges, we propose the medical decision
transformer (MeDT), a novel and versatile framework based on the
goal-conditioned reinforcement learning paradigm for sepsis treatment
recommendation. MeDT uses the decision transformer architecture to learn a
policy for drug dosage recommendation. During offline training, MeDT utilizes
collected treatment trajectories to predict administered treatments for each
time step, incorporating known treatment outcomes, target acuity scores, past
treatment decisions, and current and past medical states. This analysis enables
MeDT to capture complex dependencies among a patient's medical history,
treatment decisions, outcomes, and short-term effects on stability. Our
proposed conditioning uses acuity scores to address sparse reward issues and to
facilitate clinician-model interactions, enhancing decision-making. Following
training, MeDT can generate tailored treatment recommendations by conditioning
on the desired positive outcome (survival) and user-specified short-term
stability improvements. We carry out rigorous experiments on data from the
MIMIC-III dataset and use off-policy evaluation to demonstrate that MeDT
recommends interventions that outperform or are competitive with existing
offline reinforcement learning methods while enabling a more interpretable,
personalized and clinician-directed approach.
