  Recommending novel content, which expands user horizons by introducing them
to new interests, has been shown to improve users' long-term experience on
recommendation platforms \cite{chen2021values}. Users however are not
constantly looking to explore novel content. It is therefore crucial to
understand their novelty-seeking intent and adjust the recommendation policy
accordingly. Most existing literature models a user's propensity to choose
novel content or to prefer a more diverse set of recommendations at individual
interactions. Hierarchical structure, on the other hand, exists in a user's
novelty-seeking intent, which is manifested as a static and intrinsic user
preference for seeking novelty along with a dynamic session-based propensity.
To this end, we propose a novel hierarchical reinforcement learning-based
method to model the hierarchical user novelty-seeking intent, and to adapt the
recommendation policy accordingly based on the extracted user novelty-seeking
propensity. We further incorporate diversity and novelty-related measurement in
the reward function of the hierarchical RL (HRL) agent to encourage user
exploration \cite{chen2021values}. We demonstrate the benefits of explicitly
modeling hierarchical user novelty-seeking intent in recommendations through
extensive experiments on simulated and real-world datasets. In particular, we
demonstrate that the effectiveness of our proposed hierarchical RL-based method
lies in its ability to capture such hierarchically-structured intent. As a
result, the proposed HRL model achieves superior performance on several public
datasets, compared with state-of-art baselines.
