  An autonomous service robot should be able to interact with its environment
safely and robustly without requiring human assistance. Unstructured
environments are challenging for robots since the exact prediction of outcomes
is not always possible. Even when the robot behaviors are well-designed, the
unpredictable nature of physical robot-object interaction may prevent success
in object manipulation. Therefore, execution of a manipulation action may
result in an undesirable outcome involving accidents or damages to the objects
or environment. Situation awareness becomes important in such cases to enable
the robot to (i) maintain the integrity of both itself and the environment,
(ii) recover from failed tasks in the short term, and (iii) learn to avoid
failures in the long term. For this purpose, robot executions should be
continuously monitored, and failures should be detected and classified
appropriately. In this work, we focus on detecting and classifying both
manipulation and post-manipulation phase failures using the same exteroception
setup. We cover a diverse set of failure types for primary tabletop
manipulation actions. In order to detect these failures, we propose FINO-Net
[1], a deep multimodal sensor fusion based classifier network. Proposed network
accurately detects and classifies failures from raw sensory data without any
prior knowledge. In this work, we use our extended FAILURE dataset [1] with 99
new multimodal manipulation recordings and annotate them with their
corresponding failure types. FINO-Net achieves 0.87 failure detection and 0.80
failure classification F1 scores. Experimental results show that proposed
architecture is also appropriate for real-time use.
