  Computed Tomography (CT) scans provide detailed and accurate information of
internal structures in the body. They are constructed by sending x-rays through
the body from different directions and combining this information into a
three-dimensional volume. Such volumes can then be used to diagnose a wide
range of conditions and allow for volumetric measurements of organs. In this
work, we tackle the problem of reconstructing CT images from biplanar x-rays
only. X-rays are widely available and even if the CT reconstructed from these
radiographs is not a replacement of a complete CT in the diagnostic setting, it
might serve to spare the patients from radiation where a CT is only acquired
for rough measurements such as determining organ size. We propose a novel
method based on the transformer architecture, by framing the underlying task as
a language translation problem. Radiographs and CT images are first embedded
into latent quantized codebook vectors using two different autoencoder
networks. We then train a GPT model, to reconstruct the codebook vectors of the
CT image, conditioned on the codebook vectors of the x-rays and show that this
approach leads to realistic looking images. To encourage further research in
this direction, we make our code publicly available on GitHub: XXX.
