  With the advancement of deep learning techniques, major cloud providers and
niche machine learning service providers start to offer their cloud-based
machine learning tools, also known as machine learning as a service (MLaaS), to
the public. According to our measurement, for the same task, these MLaaSes from
different providers have varying performance due to the proprietary datasets,
models, etc. Federating different MLaaSes together allows us to improve the
analytic performance further. However, naively aggregating results from
different MLaaSes not only incurs significant momentary cost but also may lead
to sub-optimal performance gain due to the introduction of possible
false-positive results. In this paper, we propose Armol, a framework to
federate the right selection of MLaaS providers to achieve the best possible
analytic performance. We first design a word grouping algorithm to unify the
output labels across different providers. We then present a deep combinatorial
reinforcement learning based-approach to maximize the accuracy while minimizing
the cost. The predictions from the selected providers are then aggregated
together using carefully chosen ensemble strategies. The real-world
trace-driven evaluation further demonstrates that Armol is able to achieve the
same accuracy results with $67\%$ less inference cost.
