  Deep neural networks have evolved as the leading approach in 3D medical image
segmentation due to their outstanding performance. However, the ever-increasing
model size and computation cost of deep neural networks have become the primary
barrier to deploying them on real-world resource-limited hardware. In pursuit
of improving performance and efficiency, we propose a 3D medical image
segmentation model, named Efficient to Efficient Network (E2ENet),
incorporating two parametrically and computationally efficient designs. i.
Dynamic sparse feature fusion (DSFF) mechanism: it adaptively learns to fuse
informative multi-scale features while reducing redundancy. ii. Restricted
depth-shift in 3D convolution: it leverages the 3D spatial information while
keeping the model and computational complexity as 2D-based methods. We conduct
extensive experiments on BTCV, AMOS-CT and Brain Tumor Segmentation Challenge,
demonstrating that E2ENet consistently achieves a superior trade-off between
accuracy and efficiency than prior arts across various resource constraints.
E2ENet achieves comparable accuracy on the large-scale challenge AMOS-CT, while
saving over 68\% parameter count and 29\% FLOPs in the inference phase,
compared with the previous best-performing method. Our code has been made
available at: https://github.com/boqian333/E2ENet-Medical.
