  In this paper, we consider depth control problems of an autonomous underwater
vehicle (AUV) for tracking the desired depth trajectories. Due to the unknown
dynamical model of the AUV, the problems cannot be solved by most of
model-based controllers. To this purpose, we formulate the depth control
problems of the AUV as continuous-state, continuous-action Markov decision
processes (MDPs) under unknown transition probabilities. Based on deterministic
policy gradient (DPG) and neural network approximation, we propose a model-free
reinforcement learning (RL) algorithm that learns a state-feedback controller
from sampled trajectories of the AUV. To improve the performance of the RL
algorithm, we further propose a batch-learning scheme through replaying
previous prioritized trajectories. We illustrate with simulations that our
model-free method is even comparable to the model-based controllers as LQI and
NMPC. Moreover, we validate the effectiveness of the proposed RL algorithm on a
seafloor data set sampled from the South China Sea.
