  Recent works on Lottery Ticket Hypothesis have shown that pre-trained
language models (PLMs) contain smaller matching subnetworks(winning tickets)
which are capable of reaching accuracy comparable to the original models.
However, these tickets are proved to be notrobust to adversarial examples, and
even worse than their PLM counterparts. To address this problem, we propose a
novel method based on learning binary weight masks to identify robust tickets
hidden in the original PLMs. Since the loss is not differentiable for the
binary mask, we assign the hard concrete distribution to the masks and
encourage their sparsity using a smoothing approximation of L0
regularization.Furthermore, we design an adversarial loss objective to guide
the search for robust tickets and ensure that the tickets perform well bothin
accuracy and robustness. Experimental results show the significant improvement
of the proposed method over previous work on adversarial robustness evaluation.
