  We propose a novel approach to multimodal sensor fusion for Ambient Assisted
Living (AAL) which takes advantage of learning using privileged information
(LUPI). We address two major shortcomings of standard multimodal approaches,
limited area coverage and reduced reliability. Our new framework fuses the
concept of modality hallucination with triplet learning to train a model with
different modalities to handle missing sensors at inference time. We evaluate
the proposed model on inertial data from a wearable accelerometer device, using
RGB videos and skeletons as privileged modalities, and show an improvement of
accuracy of an average 6.6% on the UTD-MHAD dataset and an average 5.5% on the
Berkeley MHAD dataset, reaching a new state-of-the-art for inertial-only
classification accuracy on these datasets. We validate our framework through
several ablation studies.
