  Substantial research works have shown that deep models, e.g., pre-trained
models, on the large corpus can learn universal language representations, which
are beneficial for downstream NLP tasks. However, these powerful models are
also vulnerable to various privacy attacks, while much sensitive information
exists in the training dataset. The attacker can easily steal sensitive
information from public models, e.g., individuals' email addresses and phone
numbers. In an attempt to address these issues, particularly the unauthorized
use of private data, we introduce a novel watermarking technique via a
backdoor-based membership inference approach named TextMarker, which can
safeguard diverse forms of private information embedded in the training text
data. Specifically, TextMarker only requires data owners to mark a small number
of samples for data copyright protection under the black-box access assumption
to the target model. Through extensive evaluation, we demonstrate the
effectiveness of TextMarker on various real-world datasets, e.g., marking only
0.1% of the training dataset is practically sufficient for effective membership
inference with negligible effect on model utility. We also discuss potential
countermeasures and show that TextMarker is stealthy enough to bypass them.
