  In computer vision, traditional ensemble learning methods exhibit either a
low training efficiency or the limited performance to enhance the reliability
of deep neural networks. In this paper, we propose a lightweight,
loss-function-free, and architecture-agnostic ensemble learning by the
Decorrelating Structure via Adapters (DSA) for various visual tasks.
Concretely, the proposed DSA leverages the structure-diverse adapters to
decorrelate multiple prediction heads without any tailed regularization or
loss. This allows DSA to be easily extensible to architecture-agnostic networks
for a range of computer vision tasks. Importantly, the theoretically analysis
shows that the proposed DSA has a lower bias and variance than that of the
single head based method (which is adopted by most of the state of art
approaches). Consequently, the DSA makes deep networks reliable and robust for
the various real-world challenges, \textit{e.g.}, data corruption, and label
noises. Extensive experiments combining the proposed method with FreeMatch
achieved the accuracy improvements of 5.35% on CIFAR-10 dataset with 40 labeled
data and 0.71% on CIFAR-100 dataset with 400 labeled data. Besides, combining
the proposed method with DualPose achieved the improvements in the Percentage
of Correct Keypoints (PCK) by 2.08% on the Sniffing dataset with 100 data (30
labeled data), 5.2% on the FLIC dataset with 100 data (including 50 labeled
data), and 2.35% on the LSP dataset with 200 data (100 labeled data).
