  In real-world clinical practice, overlooking unanticipated findings can
result in serious consequences. However, supervised learning, which is the
foundation for the current success of deep learning, only encourages models to
identify abnormalities that are defined in datasets in advance. Therefore,
abnormality detection must be implemented in medical images that are not
limited to a specific disease category. In this study, we demonstrate an
unsupervised learning framework for pixel-wise abnormality detection in brain
magnetic resonance imaging captured from a patient population with metastatic
brain tumor. Our concept is as follows: If an image reconstruction network can
faithfully reproduce the global features of normal anatomy, then the abnormal
lesions in unseen images can be identified based on the local difference from
those reconstructed as normal by a discriminative network. Both networks are
trained on a dataset comprising only normal images without labels. In addition,
we devise a metric to evaluate the anatomical fidelity of the reconstructed
images and confirm that the overall detection performance is improved when the
image reconstruction network achieves a higher score. For evaluation,
clinically significant abnormalities are comprehensively segmented. The results
show that the area under the receiver operating characteristics curve values
for metastatic brain tumors, extracranial metastatic tumors, postoperative
cavities, and structural changes are 0.78, 0.61, 0.91, and 0.60, respectively.
