  Large Language Models (LLMs) raise concerns about lowering the cost of
generating texts that could be used for unethical or illegal purposes,
especially on social media. This paper investigates the promise of such models
to help enforce legal requirements related to the disclosure of sponsored
content online. We investigate the use of LLMs for generating synthetic
Instagram captions with two objectives: The first objective (fidelity) is to
produce realistic synthetic datasets. For this, we implement content-level and
network-level metrics to assess whether synthetic captions are realistic. The
second objective (utility) is to create synthetic data that is useful for
sponsored content detection. For this, we evaluate the effectiveness of the
generated synthetic data for training classifiers to identify undisclosed
advertisements on Instagram. Our investigations show that the objectives of
fidelity and utility may conflict and that prompt engineering is a useful but
insufficient strategy. Additionally, we find that while individual synthetic
posts may appear realistic, collectively they lack diversity, topic
connectivity, and realistic user interaction patterns.
