  Strategy training is a multidisciplinary rehabilitation approach that teaches
skills to reduce disability among those with cognitive impairments following a
stroke. Strategy training has been shown in randomized, controlled clinical
trials to be a more feasible and efficacious intervention for promoting
independence than traditional rehabilitation approaches. A standardized
fidelity assessment is used to measure adherence to treatment principles by
examining guided and directed verbal cues in video recordings of rehabilitation
sessions. Although the fidelity assessment for detecting guided and directed
verbal cues is valid and feasible for single-site studies, it can become labor
intensive, time consuming, and expensive in large, multi-site pragmatic trials.
To address this challenge to widespread strategy training implementation, we
leveraged natural language processing (NLP) techniques to automate the strategy
training fidelity assessment, i.e., to automatically identify guided and
directed verbal cues from video recordings of rehabilitation sessions. We
developed a rule-based NLP algorithm, a long-short term memory (LSTM) model,
and a bidirectional encoder representation from transformers (BERT) model for
this task. The best performance was achieved by the BERT model with a 0.8075
F1-score. This BERT model was verified on an external validation dataset
collected from a separate major regional health system and achieved an F1 score
of 0.8259, which shows that the BERT model generalizes well. The findings from
this study hold widespread promise in psychology and rehabilitation
intervention research and practice.
