  Machine Learning (ML) and Deep Learning (DL) innovations are being introduced
at such a rapid pace that model owners and evaluators are hard-pressed
analyzing and studying them. This is exacerbated by the complicated procedures
for evaluation. The lack of standard systems and efficient techniques for
specifying and provisioning ML/DL evaluation is the main cause of this "pain
point". This work discusses common pitfalls for replicating DL model
evaluation, and shows that these subtle pitfalls can affect both accuracy and
performance. It then proposes a solution to remedy these pitfalls called
MLModelScope, a specification for repeatable model evaluation and a runtime to
provision and measure experiments. We show that by easing the model
specification and evaluation process, MLModelScope facilitates rapid adoption
of ML/DL innovations.
