  The objective of this paper is to be able to separate a video into its
natural layers, and to control which of the separated layers to attend to. For
example, to be able to separate reflections, transparency or object motion. We
make the following three contributions: (i) we introduce a new structured
neural network architecture that explicitly incorporates layers (as spatial
masks) into its design. This improves separation performance over previous
general purpose networks for this task; (ii) we demonstrate that we can augment
the architecture to leverage external cues such as audio for controllability
and to help disambiguation; and (iii) we experimentally demonstrate the
effectiveness of our approach and training procedure with controlled
experiments while also showing that the proposed model can be successfully
applied to real-word applications such as reflection removal and action
recognition in cluttered scenes.
