  In an emergency room (ER) setting, stroke triage or screening is a common
challenge. A quick CT is usually done instead of MRI due to MRI's slow
throughput and high cost. Clinical tests are commonly referred to during the
process, but the misdiagnosis rate remains high. We propose a novel multimodal
deep learning framework, DeepStroke, to achieve computer-aided stroke presence
assessment by recognizing patterns of minor facial muscles incoordination and
speech inability for patients with suspicion of stroke in an acute setting. Our
proposed DeepStroke takes one-minute facial video data and audio data readily
available during stroke triage for local facial paralysis detection and global
speech disorder analysis. Transfer learning was adopted to reduce
face-attribute biases and improve generalizability. We leverage a multi-modal
lateral fusion to combine the low- and high-level features and provide mutual
regularization for joint training. Novel adversarial training is introduced to
obtain identity-free and stroke-discriminative features. Experiments on our
video-audio dataset with actual ER patients show that DeepStroke outperforms
state-of-the-art models and achieves better performance than both a triage team
and ER doctors, attaining a 10.94% higher sensitivity and maintaining 7.37%
higher accuracy than traditional stroke triage when specificity is aligned.
Meanwhile, each assessment can be completed in less than six minutes,
demonstrating the framework's great potential for clinical translation.
