  Reproducible research in Machine Learning has seen a salutary abundance of
progress lately: workflows, transparency, and statistical analysis of
validation and test performance. We build on these efforts and take them
further. We offer a principled experimental design methodology, based on linear
mixed models, to study and separate the effects of multiple factors of
variation in machine learning experiments. This approach allows to account for
the effects of architecture, optimizer, hyper-parameters, intentional
randomization, as well as unintended lack of determinism across reruns. We
illustrate that methodology by analyzing Matching Networks, Prototypical
Networks and TADAM on the miniImagenet dataset.
