  Using language models as a remote service entails sending private information
to an untrusted provider. In addition, potential eavesdroppers can intercept
the messages, thereby exposing the information. In this work, we explore the
prospects of avoiding such data exposure at the level of text manipulation. We
focus on text classification models, examining various token mapping and
contextualized manipulation functions in order to see whether classifier
accuracy may be maintained while keeping the original text unrecoverable. We
find that although some token mapping functions are easy and straightforward to
implement, they heavily influence performance on the downstream task, and via a
sophisticated attacker can be reconstructed. In comparison, the contextualized
manipulation provides an improvement in performance.
