  Effective network congestion control strategies are key to keeping the
Internet (or any large computer network) operational. Network congestion
control has been dominated by hand-crafted heuristics for decades. Recently,
ReinforcementLearning (RL) has emerged as an alternative to automatically
optimize such control strategies. Research so far has primarily considered RL
interfaces which block the sender while an agent considers its next action.
This is largely an artifact of building on top of frameworks designed for RL in
games (e.g. OpenAI Gym). However, this does not translate to real-world
networking environments, where a network sender waiting on a policy without
sending data leads to under-utilization of bandwidth. We instead propose to
formulate congestion control with an asynchronous RL agent that handles delayed
actions. We present MVFST-RL, a scalable framework for congestion control in
the QUIC transport protocol that leverages state-of-the-art in asynchronous RL
training with off-policy correction. We analyze modeling improvements to
mitigate the deviation from Markovian dynamics, and evaluate our method on
emulated networks from the Pantheon benchmark platform. The source code is
publicly available at https://github.com/facebookresearch/mvfst-rl.
