  Colorectal polyps are important precursors to colon cancer, the third most
common cause of cancer mortality for both men and women. It is a disease where
early detection is of crucial importance. Colonoscopy is commonly used for
early detection of cancer and precancerous pathology. It is a demanding
procedure requiring significant amount of time from specialized physicians and
nurses, in addition to a significant miss-rates of polyps by specialists.
Automated polyp detection in colonoscopy videos has been demonstrated to be a
promising way to handle this problem. {However, polyps detection is a
challenging problem due to the availability of limited amount of training data
and large appearance variations of polyps. To handle this problem, we propose a
novel deep learning method Y-Net that consists of two encoder networks with a
decoder network. Our proposed Y-Net method} relies on efficient use of
pre-trained and un-trained models with novel sum-skip-concatenation operations.
Each of the encoders are trained with encoder specific learning rate along the
decoder. Compared with the previous methods employing hand-crafted features or
2-D/3-D convolutional neural network, our approach outperforms state-of-the-art
methods for polyp detection with 7.3% F1-score and 13% recall improvement.
