  Removing soft and self shadows that lack clear boundaries from a single image
is still challenging. Self shadows are shadows that are cast on the object
itself. Most existing methods rely on binary shadow masks, without considering
the ambiguous boundaries of soft and self shadows. In this paper, we present
DeS3, a method that removes hard, soft and self shadows based on adaptive
attention and ViT similarity. Our novel ViT similarity loss utilizes features
extracted from a pre-trained Vision Transformer. This loss helps guide the
reverse sampling towards recovering scene structures. Our adaptive attention is
able to differentiate shadow regions from the underlying objects, as well as
shadow regions from the object casting the shadow. This capability enables DeS3
to better recover the structures of objects even when they are partially
occluded by shadows. Different from existing methods that rely on constraints
during the training phase, we incorporate the ViT similarity during the
sampling stage. Our method outperforms state-of-the-art methods on the SRD,
AISTD, LRSS, USR and UIUC datasets, removing hard, soft, and self shadows
robustly. Specifically, our method outperforms the SOTA method by 16\% of the
RMSE of the whole image on the LRSS dataset. Our data and code is available at:
\url{https://github.com/jinyeying/DeS3_Deshadow}
