  This paper presents a distributionally robust Q-Learning algorithm (DrQ)
which leverages Wasserstein ambiguity sets to provide idealistic probabilistic
out-of-sample safety guarantees during online learning. First, we follow past
work by separating the constraint functions from the principal objective to
create a hierarchy of machines which estimate the feasible state-action space
within the constrained Markov decision process (CMDP). DrQ works within this
framework by augmenting constraint costs with tightening offset variables
obtained through Wasserstein distributionally robust optimization (DRO). These
offset variables correspond to worst-case distributions of modeling error
characterized by the TD-errors of the constraint Q-functions. This procedure
allows us to safely approach the nominal constraint boundaries.
  Using a case study of lithium-ion battery fast charging, we explore how
idealistic safety guarantees translate to generally improved safety relative to
conventional methods.
