  User interest exploration is an important and challenging topic in
recommender systems, which alleviates the closed-loop effects between
recommendation models and user-item interactions. Contextual bandit (CB)
algorithms strive to make a good trade-off between exploration and exploitation
so that users' potential interests have chances to expose. However, classical
CB algorithms can only be applied to a small, sampled item set (usually
hundreds), which forces the typical applications in recommender systems limited
to candidate post-ranking, homepage top item ranking, ad creative selection, or
online model selection (A/B test).
  In this paper, we introduce two simple but effective hierarchical CB
algorithms to make a classical CB model (such as LinUCB and Thompson Sampling)
capable to explore users' interest in the entire item space without limiting it
to a small item set. We first construct a hierarchy item tree via a bottom-up
clustering algorithm to organize items in a coarse-to-fine manner. Then we
propose a hierarchical CB (HCB) algorithm to explore users' interest in the
hierarchy tree. HCB takes the exploration problem as a series of
decision-making processes, where the goal is to find a path from the root to a
leaf node, and the feedback will be back-propagated to all the nodes in the
path. We further propose a progressive hierarchical CB (pHCB) algorithm, which
progressively extends visible nodes which reach a confidence level for
exploration, to avoid misleading actions on upper-level nodes in the sequential
decision-making process. Extensive experiments on two public recommendation
datasets demonstrate the effectiveness and flexibility of our methods.
