  The recommendation system is an important commercial application of machine
learning, where billions of feed views in the information flow every day. In
reality, the interaction between user and item usually makes user's interest
changing over time, thus many companies (e.g. ByteDance, Baidu, Alibaba, and
Weibo) employ online learning as an effective way to quickly capture user
interests. However, hundreds of billions of model parameters present online
learning with challenges for real-time model deployment. Besides, model
stability is another key point for online learning. To this end, we design and
implement a symmetric fusion online learning system framework called WeiPS,
which integrates model training and model inference. Specifically, WeiPS
carries out second level model deployment by streaming update mechanism to
satisfy the consistency requirement. Moreover, it uses multi-level fault
tolerance and real-time domino degradation to achieve high availability
requirement.
