  We present the Human And Robot Multimodal Observations of Natural Interactive
Collaboration (HARMONIC) data set. This is a large multimodal data set of human
interactions with a robotic arm in a shared autonomy setting designed to
imitate assistive eating. The data set provides human, robot, and environmental
data views of twenty-four different people engaged in an assistive eating task
with a 6 degree-of-freedom (DOF) robot arm. From each participant, we recorded
video of both eyes, egocentric video from a head-mounted camera, joystick
commands, electromyography from the forearm used to operate the joystick, third
person stereo video, and the joint positions of the 6 DOF robot arm. Also
included are several features that come as a direct result of these recordings,
such as eye gaze projected onto the egocentric video, body pose, hand pose, and
facial keypoints. These data streams were collected specifically because they
have been shown to be closely related to human mental states and intention.
This data set could be of interest to researchers studying intention
prediction, human mental state modeling, and shared autonomy. Data streams are
provided in a variety of formats such as video and human-readable CSV and YAML
files.
