  The proliferation of healthcare data has brought the opportunities of
applying data-driven approaches, such as machine learning methods, to assist
diagnosis. Recently, many deep learning methods have been shown with impressive
successes in predicting disease status with raw input data. However, the
"black-box" nature of deep learning and the high-reliability requirement of
biomedical applications have created new challenges regarding the existence of
confounding factors. In this paper, with a brief argument that inappropriate
handling of confounding factors will lead to models' sub-optimal performance in
real-world applications, we present an efficient method that can remove the
influences of confounding factors such as age or gender to improve the
across-cohort prediction accuracy of neural networks. One distinct advantage of
our method is that it only requires minimal changes of the baseline model's
architecture so that it can be plugged into most of the existing neural
networks. We conduct experiments across CT-scan, MRA, and EEG brain wave with
convolutional neural networks and LSTM to verify the efficiency of our method.
