  The instability of myoelectric signals over time complicates their use to
control highly articulated prostheses. To address this problem, studies have
tried to combine surface electromyography with modalities that are less
affected by the amputation and environment, such as accelerometry or gaze
information. In the latter case, the hypothesis is that a subject looks at the
object he or she intends to manipulate and that knowing this object's
affordances allows to constrain the set of possible grasps. In this paper, we
develop an automated way to detect stable fixations and show that gaze
information is indeed helpful in predicting hand movements. In our multimodal
approach, we automatically detect stable gazes and segment an object of
interest around the subject's fixation in the visual frame. The patch extracted
around this object is subsequently fed through an off-the-shelf deep
convolutional neural network to obtain a high level feature representation,
which is then combined with traditional surface electromyography in the
classification stage. Tests have been performed on a dataset acquired from five
intact subjects who performed ten types of grasps on various objects as well as
in a functional setting. They show that the addition of gaze information
increases the classification accuracy considerably. Further analysis
demonstrates that this improvement is consistent for all grasps and
concentrated during the movement onset and offset.
