  To maximize cumulative user engagement (e.g. cumulative clicks) in sequential
recommendation, it is often needed to tradeoff two potentially conflicting
objectives, that is, pursuing higher immediate user engagement (e.g.,
click-through rate) and encouraging user browsing (i.e., more items exposured).
Existing works often study these two tasks separately, thus tend to result in
sub-optimal results. In this paper, we study this problem from an online
optimization perspective, and propose a flexible and practical framework to
explicitly tradeoff longer user browsing length and high immediate user
engagement. Specifically, by considering items as actions, user's requests as
states and user leaving as an absorbing state, we formulate each user's
behavior as a personalized Markov decision process (MDP), and the problem of
maximizing cumulative user engagement is reduced to a stochastic shortest path
(SSP) problem. Meanwhile, with immediate user engagement and quit probability
estimation, it is shown that the SSP problem can be efficiently solved via
dynamic programming. Experiments on real-world datasets demonstrate the
effectiveness of the proposed approach. Moreover, this approach is deployed at
a large E-commerce platform, achieved over 7% improvement of cumulative clicks.
