  We introduce a new memory architecture for navigation in previously unseen
environments, inspired by landmark-based navigation in animals. The proposed
semi-parametric topological memory (SPTM) consists of a (non-parametric) graph
with nodes corresponding to locations in the environment and a (parametric)
deep network capable of retrieving nodes from the graph based on observations.
The graph stores no metric information, only connectivity of locations
corresponding to the nodes. We use SPTM as a planning module in a navigation
system. Given only 5 minutes of footage of a previously unseen maze, an
SPTM-based navigation agent can build a topological map of the environment and
use it to confidently navigate towards goals. The average success rate of the
SPTM agent in goal-directed navigation across test environments is higher than
the best-performing baseline by a factor of three. A video of the agent is
available at https://youtu.be/vRF7f4lhswo
