  Stochastic video generation is particularly challenging when the camera is
mounted on a moving platform, as camera motion interacts with observed image
pixels, creating complex spatio-temporal dynamics and making the problem
partially observable. Existing methods typically address this by focusing on
raw pixel-level image reconstruction without explicitly modelling camera motion
dynamics. We propose a solution by considering camera motion or action as part
of the observed image state, modelling both image and action within a
multi-modal learning framework. We introduce three models: Video Generation
with Learning Action Prior (VG-LeAP) treats the image-action pair as an
augmented state generated from a single latent stochastic process and uses
variational inference to learn the image-action latent prior; Causal-LeAP,
which establishes a causal relationship between action and the observed image
frame at time $t$, learning an action prior conditioned on the observed image
states; and RAFI, which integrates the augmented image-action state concept
into flow matching with diffusion generative processes, demonstrating that this
action-conditioned image generation concept can be extended to other
diffusion-based models. We emphasize the importance of multi-modal training in
partially observable video generation problems through detailed empirical
studies on our new video action dataset, RoAM.
