  Purpose: we evaluated the generalization capability of deep neural networks
(DNNs), trained to classify chest X-rays as Covid-19, normal or pneumonia,
using a relatively small and mixed dataset. Methods: we proposed a DNN to
perform lung segmentation and classification, stacking a segmentation module
(U-Net), an original intermediate module and a classification module
(DenseNet201). To evaluate generalization, we tested the DNN with an external
dataset (from distinct localities) and used Bayesian inference to estimate
probability distributions of performance metrics. Results: our DNN achieved
0.917 AUC on the external test dataset, and a DenseNet without segmentation,
0.906. Bayesian inference indicated mean accuracy of 76.1% and [0.695, 0.826]
95% HDI (highest density interval, which concentrates 95% of the metric's
probability mass) with segmentation and, without segmentation, 71.7% and
[0.646, 0.786]. Conclusion: employing a novel DNN evaluation technique, which
uses LRP and Brixia scores, we discovered that areas where radiologists found
strong Covid-19 symptoms are the most important for the stacked DNN
classification. External validation showed smaller accuracies than internal,
indicating difficulty in generalization, which is positively affected by
segmentation. Finally, the performance in the external dataset and the analysis
with LRP suggest that DNNs can be trained in small and mixed datasets and still
successfully detect Covid-19.
