  Eye gaze estimation and simultaneous semantic understanding of a user through
eye images is a crucial component in Virtual and Mixed Reality; enabling energy
efficient rendering, multi-focal displays and effective interaction with 3D
content. In head-mounted VR/MR devices the eyes are imaged off-axis to avoid
blocking the user's gaze, this view-point makes drawing eye related inferences
very challenging. In this work, we present EyeNet, the first single deep neural
network which solves multiple heterogeneous tasks related to eye gaze
estimation and semantic user understanding for an off-axis camera setting. The
tasks include eye segmentation, blink detection, emotive expression
classification, IR LED glints detection, pupil and cornea center estimation. To
train EyeNet end-to-end we employ both hand labelled supervision and model
based supervision. We benchmark all tasks on MagicEyes, a large and new dataset
of 587 subjects with varying morphology, gender, skin-color, make-up and
imaging conditions.
