  Accurate models of clinical actions and their impacts on disease progression
are critical for estimating personalized optimal dynamic treatment regimes
(DTRs) in medical/health research, especially in managing chronic conditions.
Traditional statistical methods for DTRs usually focus on estimating the
optimal treatment or dosage at each given medical intervention, but overlook
the important question of "when this intervention should happen." We fill this
gap by developing a two-step Bayesian approach to optimize clinical decisions
with timing. In the first step, we build a generative model for a sequence of
medical interventions-which are discrete events in continuous time-with a
marked temporal point process (MTPP) where the mark is the assigned treatment
or dosage. Then this clinical action model is embedded into a Bayesian joint
framework where the other components model clinical observations including
longitudinal medical measurements and time-to-event data conditional on
treatment histories. In the second step, we propose a policy gradient method to
learn the personalized optimal clinical decision that maximizes the patient
survival by interacting the MTPP with the model on clinical observations while
accounting for uncertainties in clinical observations learned from the
posterior inference of the Bayesian joint model in the first step. A signature
application of the proposed approach is to schedule follow-up visitations and
assign a dosage at each visitation for patients after kidney transplantation.
We evaluate our approach with comparison to alternative methods on both
simulated and real-world datasets. In our experiments, the personalized
decisions made by the proposed method are clinically useful: they are
interpretable and successfully help improve patient survival.
