  Most advances in medical lesion detection network are limited to subtle
modification on the conventional detection network designed for natural images.
However, there exists a vast domain gap between medical images and natural
images where the medical image detection often suffers from several
domain-specific challenges, such as high lesion/background similarity, dominant
tiny lesions, and severe class imbalance. Is a hand-crafted detection network
tailored for natural image undoubtedly good enough over a discrepant medical
lesion domain? Is there more powerful operations, filters, and sub-networks
that better fit the medical lesion detection problem to be discovered? In this
paper, we introduce a novel ElixirNet that includes three components: 1)
TruncatedRPN balances positive and negative data for false positive reduction;
2) Auto-lesion Block is automatically customized for medical images to
incorporate relation-aware operations among region proposals, and leads to more
suitable and efficient classification and localization. 3) Relation transfer
module incorporates the semantic relationship and transfers the relevant
contextual information with an interpretable the graph thus alleviates the
problem of lack of annotations for all types of lesions. Experiments on
DeepLesion and Kits19 prove the effectiveness of ElixirNet, achieving
improvement of both sensitivity and precision over FPN with fewer parameters.
