  Several recent works show impressive results in mapping language-based human
commands and image scene observations to direct robot executable policies
(e.g., pick and place poses). However, these approaches do not consider the
uncertainty of the trained policy and simply always execute actions suggested
by the current policy as the most probable ones. This makes them vulnerable to
domain shift and inefficient in the number of required demonstrations. We
extend previous works and present the PARTNR algorithm that can detect
ambiguities in the trained policy by analyzing multiple modalities in the pick
and place poses using topological analysis. PARTNR employs an adaptive,
sensitivity-based, gating function that decides if additional user
demonstrations are required. User demonstrations are aggregated to the dataset
and used for subsequent training. In this way, the policy can adapt promptly to
domain shift and it can minimize the number of required demonstrations for a
well-trained policy. The adaptive threshold enables to achieve the
user-acceptable level of ambiguity to execute the policy autonomously and in
turn, increase the trustworthiness of our system. We demonstrate the
performance of PARTNR in a table-top pick and place task.
