  Visual foresight gives an agent a window into the future, which it can use to
anticipate events before they happen and plan strategic behavior. Although
impressive results have been achieved on video prediction in constrained
settings, these models fail to generalize when confronted with unfamiliar
real-world objects. In this paper, we tackle the generalization problem via
fast adaptation, where we train a prediction model to quickly adapt to the
observed visual dynamics of a novel object. Our method, Experience-embedded
Visual Foresight (EVF), jointly learns a fast adaptation module, which encodes
observed trajectories of the new object into a vector embedding, and a visual
prediction model, which conditions on this embedding to generate physically
plausible predictions. For evaluation, we compare our method against baselines
on video prediction and benchmark its utility on two real-world control tasks.
We show that our method is able to quickly adapt to new visual dynamics and
achieves lower error than the baselines when manipulating novel objects.
