  Embodied AI agents that search for objects in large environments such as
households often need to make efficient decisions by predicting object
locations based on partial information. We pose this as a new type of link
prediction problem: link prediction on partially observable dynamic graphs. Our
graph is a representation of a scene in which rooms and objects are nodes, and
their relationships are encoded in the edges; only parts of the changing graph
are known to the agent at each timestep. This partial observability poses a
challenge to existing link prediction approaches, which we address. We propose
a novel state representation -- Scene Graph Memory (SGM) -- with captures the
agent's accumulated set of observations, as well as a neural net architecture
called a Node Edge Predictor (NEP) that extracts information from the SGM to
search efficiently. We evaluate our method in the Dynamic House Simulator, a
new benchmark that creates diverse dynamic graphs following the semantic
patterns typically seen at homes, and show that NEP can be trained to predict
the locations of objects in a variety of environments with diverse object
movement dynamics, outperforming baselines both in terms of new scene
adaptability and overall accuracy. The codebase and more can be found at
https://www.scenegraphmemory.com.
