  Accurate medical image segmentation is critical for early medical diagnosis.
Most existing methods are based on U-shape structure and use element-wise
addition or concatenation to fuse different level features progressively in
decoder. However, both the two operations easily generate plenty of redundant
information, which will weaken the complementarity between different level
features, resulting in inaccurate localization and blurred edges of lesions. To
address this challenge, we propose a general multi-scale in multi-scale
subtraction network (M$^{2}$SNet) to finish diverse segmentation from medical
image. Specifically, we first design a basic subtraction unit (SU) to produce
the difference features between adjacent levels in encoder. Next, we expand the
single-scale SU to the intra-layer multi-scale SU, which can provide the
decoder with both pixel-level and structure-level difference information. Then,
we pyramidally equip the multi-scale SUs at different levels with varying
receptive fields, thereby achieving the inter-layer multi-scale feature
aggregation and obtaining rich multi-scale difference information. In addition,
we build a training-free network ``LossNet'' to comprehensively supervise the
task-aware features from bottom layer to top layer, which drives our
multi-scale subtraction network to capture the detailed and structural cues
simultaneously. Without bells and whistles, our method performs favorably
against most state-of-the-art methods under different evaluation metrics on
eleven datasets of four different medical image segmentation tasks of diverse
image modalities, including color colonoscopy imaging, ultrasound imaging,
computed tomography (CT), and optical coherence tomography (OCT). The source
code can be available at \url{https://github.com/Xiaoqi-Zhao-DLUT/MSNet}.
