  The widespread usage of large-scale multimodal models like CLIP has
heightened concerns about the leakage of personally identifiable information
(PII). Existing methods for identity inference in CLIP models, i.e., to detect
the presence of a person's PII used for training a CLIP model, require querying
the model with full PII, including textual descriptions of the person and
corresponding images (e.g., the name and the face photo of the person).
However, this may lead to potential privacy breach of the image, as it may have
not been seen by the target model yet. Additionally, traditional membership
inference attacks (MIAs) train shadow models to mimic the behaviors of the
target model, which incurs high computational costs, especially for large CLIP
models. To address these challenges, we propose a textual unimodal detector
(TUNI) in CLIP models, a novel method for ID inference that 1) queries the
target model with only text data; and 2) does not require training shadow
models. Firstly, we develop a feature extraction algorithm, guided by the CLIP
model, to extract features from a text description. TUNI starts with randomly
generating textual gibberish that were clearly not utilized for training, and
leverages their feature vectors to train a system of anomaly detectors. During
inference, the feature vector of each test text is fed into the anomaly
detectors to determine if the person's PII is in the training set (abnormal) or
not (normal). Moreover, TUNI can be further strengthened integrating real
images associated with the tested individuals, if available at the detector.
Extensive experiments of TUNI across various CLIP model architectures and
datasets demonstrate its superior performance over baselines, albeit with only
text data.
