  In this paper, we investigate the task of aggregating search results from
heterogeneous sources in an E-commerce environment. First, unlike traditional
aggregated web search that merely presents multi-sourced results in the first
page, this new task may present aggregated results in all pages and has to
dynamically decide which source should be presented in the current page.
Second, as pointed out by many existing studies, it is not trivial to rank
items from heterogeneous sources because the relevance scores from different
source systems are not directly comparable. To address these two issues, we
decompose the task into two subtasks in a hierarchical structure: a high-level
task for source selection where we model the sequential patterns of user
behaviors onto aggregated results in different pages so as to understand user
intents and select the relevant sources properly; and a low-level task for item
presentation where we formulate a slot filling process to sequentially present
the items instead of giving each item a relevance score when deciding the
presentation order of heterogeneous items. Since both subtasks can be naturally
formulated as sequential decision problems and learn from the future user
feedback on search results, we build our model with hierarchical reinforcement
learning. Extensive experiments demonstrate that our model obtains remarkable
improvements in search performance metrics, and achieves a higher user
satisfaction.
