  The rapid progress of Large Models (LMs) has recently revolutionized various
fields of deep learning with remarkable grades, ranging from Natural Language
Processing (NLP) to Computer Vision (CV). However, LMs are increasingly
challenged and criticized by academia and industry due to their powerful
performance but untrustworthy behavior, which urgently needs to be alleviated
by reliable methods. Despite the abundance of literature on trustworthy LMs in
NLP, a systematic survey specifically delving into the trustworthiness of LMs
in CV remains absent. In order to mitigate this gap, we summarize four relevant
concerns that obstruct the trustworthy usage in vision of LMs in this survey,
including 1) human misuse, 2) vulnerability, 3) inherent issue and 4)
interpretability. By highlighting corresponding challenge, countermeasures, and
discussion in each topic, we hope this survey will facilitate readers'
understanding of this field, promote alignment of LMs with human expectations
and enable trustworthy LMs to serve as welfare rather than disaster for human
society.
