  GenerateCT, the first approach to generating 3D medical imaging conditioned
on free-form medical text prompts, incorporates a text encoder and three key
components: a novel causal vision transformer for encoding 3D CT volumes, a
text-image transformer for aligning CT and text tokens, and a text-conditional
super-resolution diffusion model. Without directly comparable methods in 3D
medical imaging, we benchmarked GenerateCT against cutting-edge methods,
demonstrating its superiority across all key metrics. Importantly, we evaluated
GenerateCT's clinical applications in a multi-abnormality classification task.
First, we established a baseline by training a multi-abnormality classifier on
our real dataset. To further assess the model's generalization to external data
and performance with unseen prompts in a zero-shot scenario, we employed an
external set to train the classifier, setting an additional benchmark. We
conducted two experiments in which we doubled the training datasets by
synthesizing an equal number of volumes for each set using GenerateCT. The
first experiment demonstrated an 11% improvement in the AP score when training
the classifier jointly on real and generated volumes. The second experiment
showed a 7% improvement when training on both real and generated volumes based
on unseen prompts. Moreover, GenerateCT enables the scaling of synthetic
training datasets to arbitrary sizes. As an example, we generated 100,000 3D
CTs, fivefold the number in our real set, and trained the classifier
exclusively on these synthetic CTs. Impressively, this classifier surpassed the
performance of the one trained on all available real data by a margin of 8%.
Last, domain experts evaluated the generated volumes, confirming a high degree
of alignment with the text prompt. Access our code, model weights, training
data, and generated data at https://github.com/ibrahimethemhamamci/GenerateCT
