  Towards automated retinal screening, this paper makes an endeavor to
simultaneously achieve pixel-level retinal lesion segmentation and image-level
disease classification. Such a multi-task approach is crucial for accurate and
clinically interpretable disease diagnosis. Prior art is insufficient due to
three challenges, i.e., lesions lacking objective boundaries, clinical
importance of lesions irrelevant to their size, and the lack of one-to-one
correspondence between lesion and disease classes. This paper attacks the three
challenges in the context of diabetic retinopathy (DR) grading. We propose
Lesion-Net, a new variant of fully convolutional networks, with its expansive
path re-designed to tackle the first challenge. A dual Dice loss that leverages
both semantic segmentation and image classification losses is introduced to
resolve the second challenge. Lastly, we build a multi-task network that
employs Lesion-Net as a side-attention branch for both DR grading and result
interpretation. A set of 12K fundus images is manually segmented by 45
ophthalmologists for 8 DR-related lesions, resulting in 290K manual segments in
total. Extensive experiments on this large-scale dataset show that our proposed
approach surpasses the prior art for multiple tasks including lesion
segmentation, lesion classification and DR grading
