  The goal of this work is to identify the best optimizers for deep learning in
the context of cardiac image segmentation and to provide guidance on how to
design segmentation networks with effective optimization strategies. Adaptive
learning helps with fast convergence by starting with a larger learning rate
(LR) and gradually decreasing it. Momentum optimizers are particularly
effective at quickly optimizing neural networks within the accelerated schemes
category. By revealing the potential interplay between these two types of
algorithms (LR and momentum optimizers or momentum rate (MR) in short), in this
article, we explore the two variants of SGD algorithms in a single setting. We
suggest using cyclic learning as the base optimizer and integrating optimal
values of learning rate and momentum rate. We investigated the relationship of
LR and MR under an important problem of medical image segmentation of cardiac
structures from MRI and CT scans. We conducted experiments using the cardiac
imaging dataset from the ACDC challenge of MICCAI 2017, and four different
architectures shown to be successful for cardiac image segmentation problems.
Our comprehensive evaluations demonstrated that the proposed optimizer achieved
better results (over a 2\% improvement in the dice metric) than other
optimizers in deep learning literature with similar or lower computational cost
in both single and multi-object segmentation settings. We hypothesized that
combination of accelerated and adaptive optimization methods can have a drastic
effect in medical image segmentation performances. To this end, we proposed a
new cyclic optimization method (\textit{CLMR}) to address the efficiency and
accuracy problems in deep learning based medical image segmentation. The
proposed strategy yielded better generalization in comparison to adaptive
optimizers.
