  We conduct an empirical study of machine learning functionalities provided by
major cloud service providers, which we call machine learning clouds. Machine
learning clouds hold the promise of hiding all the sophistication of running
large-scale machine learning: Instead of specifying how to run a machine
learning task, users only specify what machine learning task to run and the
cloud figures out the rest. Raising the level of abstraction, however, rarely
comes free - a performance penalty is possible. How good, then, are current
machine learning clouds on real-world machine learning workloads?
  We study this question with a focus on binary classication problems. We
present mlbench, a novel benchmark constructed by harvesting datasets from
Kaggle competitions. We then compare the performance of the top winning code
available from Kaggle with that of running machine learning clouds from both
Azure and Amazon on mlbench. Our comparative study reveals the strength and
weakness of existing machine learning clouds and points out potential future
directions for improvement.
