  In the era of big data, access to abundant data is crucial for driving
research forward. However, such data is often inaccessible due to privacy
concerns or high costs, particularly in healthcare domain. Generating synthetic
(tabular) data can address this, but existing models typically require
substantial amounts of data to train effectively, contradicting our objective
to solve data scarcity. To address this challenge, we propose a novel framework
to generate synthetic tabular data, powered by large language models (LLMs)
that emulates the architecture of a Generative Adversarial Network (GAN). By
incorporating data generation process as contextual information and utilizing
LLM as the optimizer, our approach significantly enhance the quality of
synthetic data generation in common scenarios with small sample sizes. Our
experimental results on public and private datasets demonstrate that our model
outperforms several state-of-art models regarding generating higher quality
synthetic data for downstream tasks while keeping privacy of the real data.
