  Robots have been increasingly better at doing tasks for humans by learning
from their feedback, but still often suffer from model misalignment due to
missing or incorrectly learned features. When the features the robot needs to
learn to perform its task are missing or do not generalize well to new
settings, the robot will not be able to learn the task the human wants and,
even worse, may learn a completely different and undesired behavior. Prior work
shows how the robot can detect when its representation is missing some feature
and can, thus, ask the human to be taught about the new feature; however, these
works do not differentiate between features that are completely missing and
those that exist but do not generalize to new environments. In the latter case,
the robot would detect misalignment and simply learn a new feature, leading to
an arbitrarily growing feature representation that can, in turn, lead to
spurious correlations and incorrect learning down the line. In this work, we
propose separating the two sources of misalignment: we propose a framework for
determining whether a feature the robot needs is incorrectly learned and does
not generalize to new environment setups vs. is entirely missing from the
robot's representation. Once we detect the source of error, we show how the
human can initiate the realignment process for the model: if the feature is
missing, we follow prior work for learning new features; however, if the
feature exists but does not generalize, we use data augmentation to expand its
training and, thus, complete the correction. We demonstrate the proposed
approach in experiments with a simulated 7DoF robot manipulator and physical
human corrections.
