  The current practice for assessing neonatal postoperative pain relies on
bedside caregivers. This practice is subjective, inconsistent, slow, and
discontinuous. To develop a reliable medical interpretation, several automated
approaches have been proposed to enhance the current practice. These approaches
are unimodal and focus mainly on assessing neonatal procedural (acute) pain. As
pain is a multimodal emotion that is often expressed through multiple
modalities, the multimodal assessment of pain is necessary especially in case
of postoperative (acute prolonged) pain. Additionally, spatio-temporal analysis
is more stable over time and has been proven to be highly effective at
minimizing misclassification errors. In this paper, we present a novel
multimodal spatio-temporal approach that integrates visual and vocal signals
and uses them for assessing neonatal postoperative pain. We conduct
comprehensive experiments to investigate the effectiveness of the proposed
approach. We compare the performance of the multimodal and unimodal
postoperative pain assessment, and measure the impact of temporal information
integration. The experimental results, on a real-world dataset, show that the
proposed multimodal spatio-temporal approach achieves the highest AUC (0.87)
and accuracy (79%), which are on average 6.67% and 6.33% higher than unimodal
approaches. The results also show that the integration of temporal information
markedly improves the performance as compared to the non-temporal approach as
it captures changes in the pain dynamic. These results demonstrate that the
proposed approach can be used as a viable alternative to manual assessment,
which would tread a path toward fully automated pain monitoring in clinical
settings, point-of-care testing, and homes.
