  Many data analytic systems have adopted a newly emerging compute resource,
serverless (SL), to handle data analytics queries in a timely and
cost-efficient manner, i.e., serverless data analytics. While these systems can
start processing queries quickly thanks to the agility and scalability of SL,
they may encounter performance- and cost-bottlenecks based on workloads due to
SL's worse performance and more expensive cost than traditional compute
resources, e.g., virtual machine (VM). In this project, we introduce Smartpick,
a SL-enabled scalable data analytics system that exploits SL and VM together to
realize composite benefits, i.e., agility from SL and better performance with
reduced cost from VM. Smartpick uses a machine learning prediction scheme,
decision-tree based Random Forest with Bayesian Optimizer, to determine SL and
VM configurations, i.e., how many SL and VM instances for queries, that meet
cost-performance goals. Smartpick offers a knob for applications to allow them
to explore a richer cost-performance tradeoff space opened by exploiting SL and
VM together. To maximize the benefits of SL, Smartpick supports a simple but
strong mechanism, called relay-instances. Smartpick also supports event-driven
prediction model retraining to deal with workload dynamics. A Smartpick
prototype was implemented on Spark and deployed on live test-beds, Amazon AWS
and Google Cloud Platform. Evaluation results indicate 97.05% and 83.49%
prediction accuracies respectively with up to 50% cost reduction as opposed to
the baselines. The results also confirm that Smartpick allows data analytics
applications to navigate the richer cost-performance tradeoff space efficiently
and to handle workload dynamics effectively and automatically.
