  Blood Glucose (BG) control involves keeping an individual's BG within a
healthy range through extracorporeal insulin injections is an important task
for people with type 1 diabetes. However,traditional patient self-management is
cumbersome and risky. Recent research has been devoted to exploring
individualized and automated BG control approaches, among which Deep
Reinforcement Learning (DRL) shows potential as an emerging approach. In this
paper, we use an exponential decay model of drug concentration to convert the
formalization of the BG control problem, which takes into account the delay and
prolongedness of drug effects, from a PAE-POMDP (Prolonged Action
Effect-Partially Observable Markov Decision Process) to a MDP, and we propose a
novel multi-step DRL-based algorithm to solve the problem. The Prioritized
Experience Replay (PER) sampling method is also used in it. Compared to
single-step bootstrapped updates, multi-step learning is more efficient and
reduces the influence from biasing targets. Our proposed method converges
faster and achieves higher cumulative rewards compared to the benchmark in the
same training environment, and improves the time-in-range (TIR), the percentage
of time the patient's BG is within the target range, in the evaluation phase.
Our work validates the effectiveness of multi-step reinforcement learning in BG
control, which may help to explore the optimal glycemic control measure and
improve the survival of diabetic patients.
