  2D/3D image registration to align a 3D volume and 2D X-ray images is a
challenging problem due to its ill-posed nature and various artifacts presented
in 2D X-ray images. In this paper, we propose a multi-agent system with an auto
attention mechanism for robust and efficient 2D/3D image registration.
Specifically, an individual agent is trained with dilated Fully Convolutional
Network (FCN) to perform registration in a Markov Decision Process (MDP) by
observing a local region, and the final action is then taken based on the
proposals from multiple agents and weighted by their corresponding confidence
levels. The contributions of this paper are threefold. First, we formulate
2D/3D registration as a MDP with observations, actions, and rewards properly
defined with respect to X-ray imaging systems. Second, to handle various
artifacts in 2D X-ray images, multiple local agents are employed efficiently
via FCN-based structures, and an auto attention mechanism is proposed to favor
the proposals from regions with more reliable visual cues. Third, a dilated
FCN-based training mechanism is proposed to significantly reduce the Degree of
Freedom in the simulation of registration environment, and drastically improve
training efficiency by an order of magnitude compared to standard CNN-based
training method. We demonstrate that the proposed method achieves high
robustness on both spine cone beam Computed Tomography data with a low
signal-to-noise ratio and data from minimally invasive spine surgery where
severe image artifacts and occlusions are presented due to metal screws and
guide wires, outperforming other state-of-the-art methods (single agent-based
and optimization-based) by a large margin.
