  Recent work has demonstrated that natural language processing techniques can
support consumer protection by automatically detecting unfair clauses in the
Terms of Service (ToS) Agreement. This work demonstrates that transformer-based
ToS analysis systems are vulnerable to adversarial attacks. We conduct
experiments attacking an unfair-clause detector with universal adversarial
triggers. Experiments show that a minor perturbation of the text can
considerably reduce the detection performance. Moreover, to measure the
detectability of the triggers, we conduct a detailed human evaluation study by
collecting both answer accuracy and response time from the participants. The
results show that the naturalness of the triggers remains key to tricking
readers.
