  Learning generalizeable policies from visual input in the presence of visual
distractions is a challenging problem in reinforcement learning. Recently,
there has been renewed interest in bisimulation metrics as a tool to address
this issue; these metrics can be used to learn representations that are, in
principle, invariant to irrelevant distractions by measuring behavioural
similarity between states. An accurate, unbiased, and scalable estimation of
these metrics has proved elusive in continuous state and action scenarios. We
propose entangled bisimulation, a bisimulation metric that allows the
specification of the distance function between states, and can be estimated
without bias in continuous state and action spaces. We show how entangled
bisimulation can meaningfully improve over previous methods on the Distracting
Control Suite (DCS), even when added on top of data augmentation techniques.
