  Powerful deep learning tools, such as convolutional neural networks (CNN),
are able to learn the input-output relationships of large complicated systems
directly from data. Encoder-decoder deep CNNs are able to extract features
directly from images, mix them with scalar inputs within a general
low-dimensional latent space, and then generate new complex 2D outputs which
represent complex physical phenomenon. One important challenge faced by deep
learning methods is large non-stationary systems whose characteristics change
quickly with time for which re-training is not feasible. In this paper we
present a method for adaptive tuning of the low-dimensional latent space of
deep encoder-decoder style CNNs based on real-time feedback to quickly
compensate for unknown and fast distribution shifts. We demonstrate our
approach for predicting the properties of a time-varying charged particle beam
in a particle accelerator whose components (accelerating electric fields and
focusing magnetic fields) are also quickly changing with time.
