  We present a novel deep learning framework named the Iteratively Optimized
Patch Label Inference Network (IOPLIN) for automatically detecting various
pavement distresses that are not solely limited to specific ones, such as
cracks and potholes. IOPLIN can be iteratively trained with only the image
label via the Expectation-Maximization Inspired Patch Label Distillation
(EMIPLD) strategy, and accomplish this task well by inferring the labels of
patches from the pavement images. IOPLIN enjoys many desirable properties over
the state-of-the-art single branch CNN models such as GoogLeNet and
EfficientNet. It is able to handle images in different resolutions, and
sufficiently utilize image information particularly for the high-resolution
ones, since IOPLIN extracts the visual features from unrevised image patches
instead of the resized entire image. Moreover, it can roughly localize the
pavement distress without using any prior localization information in the
training phase. In order to better evaluate the effectiveness of our method in
practice, we construct a large-scale Bituminous Pavement Disease Detection
dataset named CQU-BPDD consisting of 60,059 high-resolution pavement images,
which are acquired from different areas at different times. Extensive results
on this dataset demonstrate the superiority of IOPLIN over the state-of-the-art
image classification approaches in automatic pavement distress detection. The
source codes of IOPLIN are released on
\url{https://github.com/DearCaat/ioplin}, and the CQU-BPDD dataset is able to
be accessed on \url{https://dearcaat.github.io/CQU-BPDD/}.
