  Recent advances in automated skin cancer diagnosis have yielded performance
on par with board-certified dermatologists. However, these approaches
formulated skin cancer diagnosis as a simple classification task, dismissing
the potential benefit from lesion segmentation. We argue that an accurate
lesion segmentation can supplement the classification task with additive lesion
information, such as asymmetry, border, intensity, and physical size; in turn,
a faithful lesion classification can support the segmentation task with
discriminant lesion features. To this end, this paper proposes a new multi-task
framework, named MT-TransUNet, which is capable of segmenting and classifying
skin lesions collaboratively by mediating multi-task tokens in Transformers.
Furthermore, we have introduced dual-task and attended region consistency
losses to take advantage of those images without pixel-level annotation,
ensuring the model's robustness when it encounters the same image with an
account of augmentation. Our MT-TransUNet exceeds the previous state of the art
for lesion segmentation and classification tasks in ISIC-2017 and PH2; more
importantly, it preserves compelling computational efficiency regarding model
parameters (48M~vs.~130M) and inference speed (0.17s~vs.~2.02s per image). Code
will be available at https://github.com/JingyeChen/MT-TransUNet.
