  Algorithmic detection of facial palsy offers the potential to improve current
practices, which usually involve labor-intensive and subjective assessment by
clinicians. In this paper, we present a multimodal fusion-based deep learning
model that utilizes unstructured data (i.e. an image frame with facial line
segments) and structured data (i.e. features of facial expressions) to detect
facial palsy. We then contribute to a study to analyze the effect of different
data modalities and the benefits of a multimodal fusion-based approach using
videos of 21 facial palsy patients. Our experimental results show that among
various data modalities (i.e. unstructured data - RGB images and images of
facial line segments and structured data - coordinates of facial landmarks and
features of facial expressions), the feed-forward neural network using features
of facial expression achieved the highest precision of 76.22 while the
ResNet-based model using images of facial line segments achieved the highest
recall of 83.47. When we leveraged both images of facial line segments and
features of facial expressions, our multimodal fusion-based deep learning model
slightly improved the precision score to 77.05 at the expense of a decrease in
the recall score.
