  We present a transformer decoder based sports simulation engine, SportsNGEN,
trained on sports player and ball tracking sequences, that is capable of
generating sustained gameplay and accurately mimicking the decision making of
real players. By training on a large database of professional tennis tracking
data, we demonstrate that simulations produced by SportsNGEN can be used to
predict the outcomes of rallies, determine the best shot choices at any point,
and evaluate counterfactual or what if scenarios to inform coaching decisions
and elevate broadcast coverage. By combining the generated simulations with a
shot classifier and logic to start and end rallies, the system is capable of
simulating an entire tennis match. We evaluate SportsNGEN by comparing
statistics of the simulations with those of real matches between the same
players. We show that the model output sampling parameters are crucial to
simulation realism and that SportsNGEN is probabilistically well-calibrated to
real data. In addition, a generic version of SportsNGEN can be customized to a
specific player by fine-tuning on the subset of match data that includes that
player. Finally, we show qualitative results indicating the same approach works
for football.
