  Artificial intelligence has emerged as promising tool to decode a phase image
transmitted through a multimode fiber (MMF) by applying deep learning
techniques. By transmitting tens of thousands of images through the MMF, deep
neural networks (DNNs) are capable of learning how to decipher the seemingly
random output speckle patterns and unveil the intrinsic input-output
relationship. High fidelity reconstruction is obtained for datasets with a
large degree of homogeneity, which underutilizes the capacity of the combined
MMF-DNN system. Here, we show that holographic modulation can be employed to
encode an additional layer of variance on the output speckle pattern, improving
the overall transmissive capabilities of the system. Operatively we have
implemented this by adding a holographic label to the original dataset and
injecting the resulting phase image into the fiber facet through a Fourier
transform lens. The resulting speckle pattern dataset can be clustered
primarily by holographic label (rather than the image data), and can be
reconstructed without loss of fidelity. As an application, we describe how
colour images may be segmented into RGB components and each colour component
may then be labelled by distinct hologram. A UNET architecture was then used to
decode each class of speckle patterns and reconstruct the colour image without
the need for temporal synchronisation between sender and receiver
