  With the ability to generate high-quality images, text-to-image (T2I) models
can be exploited for creating inappropriate content. To prevent misuse,
existing safety measures are either based on text blacklists, which can be
easily circumvented, or harmful content classification, requiring large
datasets for training and offering low flexibility. Hence, we propose Latent
Guard, a framework designed to improve safety measures in text-to-image
generation. Inspired by blacklist-based approaches, Latent Guard learns a
latent space on top of the T2I model's text encoder, where it is possible to
check the presence of harmful concepts in the input text embeddings. Our
proposed framework is composed of a data generation pipeline specific to the
task using large language models, ad-hoc architectural components, and a
contrastive learning strategy to benefit from the generated data. The
effectiveness of our method is verified on three datasets and against four
baselines. Code and data will be shared at
https://github.com/rt219/LatentGuard.
