  Incorporating unstructured data into physical models is a challenging problem
that is emerging in data assimilation. Traditional approaches focus on
well-defined observation operators whose functional forms are typically assumed
to be known. This prevents these methods from achieving a consistent model-data
synthesis in configurations where the mapping from data-space to model-space is
unknown. To address these shortcomings, in this paper we develop a
physics-informed dynamical variational autoencoder ($\Phi$-DVAE) to embed
diverse data streams into time-evolving physical systems described by
differential equations. Our approach combines a standard, possibly nonlinear,
filter for the latent state-space model and a VAE, to assimilate the
unstructured data into the latent dynamical system. Unstructured data, in our
example systems, comes in the form of video data and velocity field
measurements, however the methodology is suitably generic to allow for
arbitrary unknown observation operators. A variational Bayesian framework is
used for the joint estimation of the encoding, latent states, and unknown
system parameters. To demonstrate the method, we provide case studies with the
Lorenz-63 ordinary differential equation, and the advection and Korteweg-de
Vries partial differential equations. Our results, with synthetic data, show
that $\Phi$-DVAE provides a data efficient dynamics encoding methodology which
is competitive with standard approaches. Unknown parameters are recovered with
uncertainty quantification, and unseen data are accurately predicted.
