  Real-world settings often do not allow acquisition of high-resolution
volumetric images for accurate morphological assessment and diagnostic. In
clinical practice it is frequently common to acquire only sparse data (e.g.
individual slices) for initial diagnostic decision making. Thereby, physicians
rely on their prior knowledge (or mental maps) of the human anatomy to
extrapolate the underlying 3D information. Accurate mental maps require years
of anatomy training, which in the first instance relies on normative learning,
i.e. excluding pathology. In this paper, we leverage Bayesian Deep Learning and
environment mapping to generate full volumetric anatomy representations from
none to a small, sparse set of slices. We evaluate proof of concept
implementations based on Generative Query Networks (GQN) and Conditional BRUNO
using abdominal CT and brain MRI as well as in a clinical application involving
sparse, motion-corrupted MR acquisition for fetal imaging. Our approach allows
to reconstruct 3D volumes from 1 to 4 tomographic slices, with a SSIM of 0.7+
and cross-correlation of 0.8+ compared to the 3D ground truth.
