  We introduce a new memory architecture, Bayesian Relational Memory (BRM), to
improve the generalization ability for semantic visual navigation agents in
unseen environments, where an agent is given a semantic target to navigate
towards. BRM takes the form of a probabilistic relation graph over semantic
entities (e.g., room types), which allows (1) capturing the layout prior from
training environments, i.e., prior knowledge, (2) estimating posterior layout
at test time, i.e., memory update, and (3) efficient planning for navigation,
altogether. We develop a BRM agent consisting of a BRM module for producing
sub-goals and a goal-conditioned locomotion module for control. When testing in
unseen environments, the BRM agent outperforms baselines that do not explicitly
utilize the probabilistic relational memory structure
