  In this study, we explore the impact of relaxing data consistency in parallel
machine learning training during a failure using various parameter server
configurations. Our failure recovery strategies include traditional
checkpointing, chain replication (which ensures a backup server takes over in
case of failure), and a novel stateless parameter server approach. In the
stateless approach, workers continue generating gradient updates even if the
parameter server is down, applying these updates once the server is back
online. We compare these techniques to a standard checkpointing approach, where
the training job is resumed from the latest checkpoint.
  To assess the resilience and performance of each configuration, we
intentionally killed the parameter server during training for each experiment.
Our experiment results indicate that the stateless parameter server approach
continues to train towards convergence and improves accuracy as much as 10\% in
the face of a failure despite using stale weights and gradients. The chain
replication and checkpointing techniques demonstrate convergence but suffer
from setbacks in accuracy due to restarting from old checkpoints. These results
suggest that allowing workers to continue generating updates during server
downtime and applying these updates later can effectively improve hardware
utilization. Furthermore, despite higher resource usage, the stateless
parameter server method incurs similar monetary costs in terms of hardware
usage compared to standard checkpointing methods due to the pricing structure
of common cloud providers.
