  While language models have made many milestones in text inference and
classification tasks, they remain susceptible to adversarial attacks that can
lead to unforeseen outcomes. Existing works alleviate this problem by equipping
language models with defense patches. However, these defense strategies often
rely on impractical assumptions or entail substantial sacrifices in model
performance. Consequently, enhancing the resilience of the target model using
such defense mechanisms is a formidable challenge. This paper introduces an
innovative model for robust text inference and classification, built upon
diffusion models (ROIC-DM). Benefiting from its training involving denoising
stages, ROIC-DM inherently exhibits greater robustness compared to conventional
language models. Moreover, ROIC-DM can attain comparable, and in some cases,
superior performance to language models, by effectively incorporating them as
advisory components. Extensive experiments conducted with several strong
textual adversarial attacks on three datasets demonstrate that (1) ROIC-DM
outperforms traditional language models in robustness, even when the latter are
fortified with advanced defense mechanisms; (2) ROIC-DM can achieve comparable
and even better performance than traditional language models by using them as
advisors.
