  The learn-from-observation (LfO) paradigm is a human-inspired mode for a
robot to learn to perform a task simply by watching it being performed. LfO can
facilitate robot integration on factory floors by minimizing disruption and
reducing tedious programming. A key component of the LfO pipeline is a
transformation of the depth camera frames to the corresponding task state and
action pairs, which are then relayed to learning techniques such as imitation
or inverse reinforcement learning for understanding the task parameters. While
several existing computer vision models analyze videos for activity
recognition, SA-Net specifically targets robotic LfO from RGB-D data. However,
SA-Net and many other models analyze frame data captured from a single
viewpoint. Their analysis is therefore highly sensitive to occlusions of the
observed task, which are frequent in deployments. An obvious way of reducing
occlusions is to simultaneously observe the task from multiple viewpoints and
synchronously fuse the multiple streams in the model. Toward this, we present
multi-view SA-Net, which generalizes the SA-Net model to allow the perception
of multiple viewpoints of the task activity, integrate them, and better
recognize the state and action in each frame. Performance evaluations on two
distinct domains establish that MVSA-Net recognizes the state-action pairs
under occlusion more accurately compared to single-view MVSA-Net and other
baselines. Our ablation studies further evaluate its performance under
different ambient conditions and establish the contribution of the architecture
components. As such, MVSA-Net offers a significantly more robust and deployable
state-action trajectory generation compared to previous methods.
