  Robot assistants for older adults and people with disabilities need to
interact with their users in collaborative tasks. The core component of these
systems is an interaction manager whose job is to observe and assess the task,
and infer the state of the human and their intent to choose the best course of
action for the robot. Due to the sparseness of the data in this domain, the
policy for such multi-modal systems is often crafted by hand; as the complexity
of interactions grows this process is not scalable. In this paper, we propose a
reinforcement learning (RL) approach to learn the robot policy. In contrast to
the dialog systems, our agent is trained with a simulator developed by using
human data and can deal with multiple modalities such as language and physical
actions. We conducted a human study to evaluate the performance of the system
in the interaction with a user. Our designed system shows promising preliminary
results when it is used by a real user.
