  Automatic assessment of impairment and disease severity is a key challenge in
data-driven medicine. We propose a novel framework to address this challenge,
which leverages AI models trained exclusively on healthy individuals. The
COnfidence-Based chaRacterization of Anomalies (COBRA) score exploits the
decrease in confidence of these models when presented with impaired or diseased
patients to quantify their deviation from the healthy population. We applied
the COBRA score to address a key limitation of current clinical evaluation of
upper-body impairment in stroke patients. The gold-standard Fugl-Meyer
Assessment (FMA) requires in-person administration by a trained assessor for
30-45 minutes, which restricts monitoring frequency and precludes physicians
from adapting rehabilitation protocols to the progress of each patient. The
COBRA score, computed automatically in under one minute, is shown to be
strongly correlated with the FMA on an independent test cohort for two
different data modalities: wearable sensors ($\rho = 0.845$, 95% CI
[0.743,0.908]) and video ($\rho = 0.746$, 95% C.I [0.594, 0.847]). To
demonstrate the generalizability of the approach to other conditions, the COBRA
score was also applied to quantify severity of knee osteoarthritis from
magnetic-resonance imaging scans, again achieving significant correlation with
an independent clinical assessment ($\rho = 0.644$, 95% C.I [0.585,0.696]).
