  Fast and accurate detection of the disease can significantly help in reducing
the strain on the healthcare facility of any country to reduce the mortality
during any pandemic. The goal of this work is to create a multimodal system
using a novel machine learning framework that uses both Chest X-ray (CXR)
images and clinical data to predict severity in COVID-19 patients. In addition,
the study presents a nomogram-based scoring technique for predicting the
likelihood of death in high-risk patients. This study uses 25 biomarkers and
CXR images in predicting the risk in 930 COVID-19 patients admitted during the
first wave of COVID-19 (March-June 2020) in Italy. The proposed multimodal
stacking technique produced the precision, sensitivity, and F1-score, of
89.03%, 90.44%, and 89.03%, respectively to identify low or high-risk patients.
This multimodal approach improved the accuracy by 6% in comparison to the CXR
image or clinical data alone. Finally, nomogram scoring system using
multivariate logistic regression -- was used to stratify the mortality risk
among the high-risk patients identified in the first stage. Lactate
Dehydrogenase (LDH), O2 percentage, White Blood Cells (WBC) Count, Age, and
C-reactive protein (CRP) were identified as useful predictor using random
forest feature selection model. Five predictors parameters and a CXR image
based nomogram score was developed for quantifying the probability of death and
categorizing them into two risk groups: survived (<50%), and death (>=50%),
respectively. The multi-modal technique was able to predict the death
probability of high-risk patients with an F1 score of 92.88 %. The area under
the curves for the development and validation cohorts are 0.981 and 0.939,
respectively.
