  Production Machine Learning involves continuous training: hosting multiple
versions of models over time, often with many model versions running at once.
When model performance does not meet expectations, Machine Learning Engineers
(MLEs) debug issues by exploring and analyzing numerous prior versions of code
and training data to identify root causes and mitigate problems. Traditional
debugging and logging tools often fall short in managing this experimental,
multi-version context. FlorDB introduces Multiversion Hindsight Logging, which
allows engineers to use the most recent version's logging statements to query
past versions, even when older versions logged different data. Log statement
propagation enables consistent injection of logging statements into past code
versions, regardless of changes to the codebase. Once log statements are
propagated across code versions, the remaining challenge in Multiversion
Hindsight Logging is to efficiently replay the new log statements based on
checkpoints from previous runs. Finally, a coherent user experience is required
to help MLEs debug across all versions of code and data. To this end, FlorDB
presents a unified relational model for efficient handling of historical
queries, offering a comprehensive view of the log history to simplify the
exploration of past code iterations. We present a performance evaluation on
diverse benchmarks confirming its scalability and the ability to deliver
real-time query responses, leveraging query-based filtering and
checkpoint-based parallelism for efficient replay.
