  Domestic service robots offer a solution to the increasing demand for daily
care and support. A human-in-the-loop approach that combines automation and
operator intervention is considered to be a realistic approach to their use in
society. Therefore, we focus on the task of retrieving target objects from
open-vocabulary user instructions in a human-in-the-loop setting, which we
define as the learning-to-rank physical objects (LTRPO) task. For example,
given the instruction "Please go to the dining room which has a round table.
Pick up the bottle on it," the model is required to output a ranked list of
target objects that the operator/user can select. In this paper, we propose
MultiRankIt, which is a novel approach for the LTRPO task. MultiRankIt
introduces the Crossmodal Noun Phrase Encoder to model the relationship between
phrases that contain referring expressions and the target bounding box, and the
Crossmodal Region Feature Encoder to model the relationship between the target
object and multiple images of its surrounding contextual environment.
Additionally, we built a new dataset for the LTRPO task that consists of
instructions with complex referring expressions accompanied by real indoor
environmental images that feature various target objects. We validated our
model on the dataset and it outperformed the baseline method in terms of the
mean reciprocal rank and recall@k. Furthermore, we conducted physical
experiments in a setting where a domestic service robot retrieved everyday
objects in a standardized domestic environment, based on users' instruction in
a human--in--the--loop setting. The experimental results demonstrate that the
success rate for object retrieval achieved 80%. Our code is available at
https://github.com/keio-smilab23/MultiRankIt.
