  In modern internet industries, deep learning based recommender systems have
became an indispensable building block for a wide spectrum of applications,
such as search engine, news feed, and short video clips. However, it remains
challenging to carry the well-trained deep models for online real-time
inference serving, with respect to the time-varying web-scale traffics from
billions of users, in a cost-effective manner. In this work, we present JIZHI -
a Model-as-a-Service system - that per second handles hundreds of millions of
online inference requests to huge deep models with more than trillions of
sparse parameters, for over twenty real-time recommendation services at Baidu,
Inc. In JIZHI, the inference workflow of every recommendation request is
transformed to a Staged Event-Driven Pipeline (SEDP), where each node in the
pipeline refers to a staged computation or I/O intensive task processor. With
traffics of real-time inference requests arrived, each modularized processor
can be run in a fully asynchronized way and managed separately. Besides, JIZHI
introduces heterogeneous and hierarchical storage to further accelerate the
online inference process by reducing unnecessary computations and potential
data access latency induced by ultra-sparse model parameters. Moreover, an
intelligent resource manager has been deployed to maximize the throughput of
JIZHI over the shared infrastructure by searching the optimal resource
allocation plan from historical logs and fine-tuning the load shedding policies
over intermediate system feedback. Extensive experiments have been done to
demonstrate the advantages of JIZHI from the perspectives of end-to-end service
latency, system-wide throughput, and resource consumption. JIZHI has helped
Baidu saved more than ten million US dollars in hardware and utility costs
while handling 200% more traffics without sacrificing inference efficiency.
