  Many measurements or observations in computer vision and machine learning
manifest as non-Euclidean data. While recent proposals (like spherical CNN)
have extended a number of deep neural network architectures to manifold-valued
data, and this has often provided strong improvements in performance, the
literature on generative models for manifold data is quite sparse. Partly due
to this gap, there are also no modality transfer/translation models for
manifold-valued data whereas numerous such methods based on generative models
are available for natural images. This paper addresses this gap, motivated by a
need in brain imaging -- in doing so, we expand the operating range of certain
generative models (as well as generative models for modality transfer) from
natural images to images with manifold-valued measurements. Our main result is
the design of a two-stream version of GLOW (flow-based invertible generative
models) that can synthesize information of a field of one type of
manifold-valued measurements given another. On the theoretical side, we
introduce three kinds of invertible layers for manifold-valued data, which are
not only analogous to their functionality in flow-based generative models
(e.g., GLOW) but also preserve the key benefits (determinants of the Jacobian
are easy to calculate). For experiments, on a large dataset from the Human
Connectome Project (HCP), we show promising results where we can reliably and
accurately reconstruct brain images of a field of orientation distribution
functions (ODF) from diffusion tensor images (DTI), where the latter has a
$5\times$ faster acquisition time but at the expense of worse angular
resolution.
