  Weakly supervised semantic segmentation (WSSS) models relying on class
activation maps (CAMs) have achieved desirable performance comparing to the
non-CAMs-based counterparts. However, to guarantee WSSS task feasible, we need
to generate pseudo labels by expanding the seeds from CAMs which is complex and
time-consuming, thus hindering the design of efficient end-to-end
(single-stage) WSSS approaches. To tackle the above dilemma, we resort to the
off-the-shelf and readily accessible saliency maps for directly obtaining
pseudo labels given the image-level class labels. Nevertheless, the salient
regions may contain noisy labels and cannot seamlessly fit the target objects,
and saliency maps can only be approximated as pseudo labels for simple images
containing single-class objects. As such, the achieved segmentation model with
these simple images cannot generalize well to the complex images containing
multi-class objects. To this end, we propose an end-to-end multi-granularity
denoising and bidirectional alignment (MDBA) model, to alleviate the noisy
label and multi-class generalization issues. Specifically, we propose the
online noise filtering and progressive noise detection modules to tackle
image-level and pixel-level noise, respectively. Moreover, a bidirectional
alignment mechanism is proposed to reduce the data distribution gap at both
input and output space with simple-to-complex image synthesis and
complex-to-simple adversarial learning. MDBA can reach the mIoU of 69.5\% and
70.2\% on validation and test sets for the PASCAL VOC 2012 dataset. The source
codes and models have been made available at
\url{https://github.com/NUST-Machine-Intelligence-Laboratory/MDBA}.
