  Object search is a challenging task because when given complex language
descriptions (e.g., "find the white cup on the table"), the robot must move its
camera through the environment and recognize the described object. Previous
works map language descriptions to a set of fixed object detectors with
predetermined noise models, but these approaches are challenging to scale
because new detectors need to be made for each object. In this work, we bridge
the gap in realistic object search by posing the search problem as a partially
observable Markov decision process (POMDP) where the object detector and visual
sensor noise in the observation model is determined by a single Deep Neural
Network conditioned on complex language descriptions. We incorporate the neural
network's outputs into our language-conditioned observation model (LCOM) to
represent dynamically changing sensor noise. With an LCOM, any language
description of an object can be used to generate an appropriate object detector
and noise model, and training an LCOM only requires readily available
supervised image-caption datasets. We empirically evaluate our method by
comparing against a state-of-the-art object search algorithm in simulation, and
demonstrate that planning with our observation model yields a significantly
higher average task completion rate (from 0.46 to 0.66) and more efficient and
quicker object search than with a fixed-noise model. We demonstrate our method
on a Boston Dynamics Spot robot, enabling it to handle complex natural language
object descriptions and efficiently find objects in a room-scale environment.
