  Robust perception relies on both bottom-up and top-down signals. Bottom-up
signals consist of what's directly observed through sensation. Top-down signals
consist of beliefs and expectations based on past experience and short-term
memory, such as how the phrase `peanut butter and~...' will be completed. The
optimal combination of bottom-up and top-down information remains an open
question, but the manner of combination must be dynamic and both context and
task dependent. To effectively utilize the wealth of potential top-down
information available, and to prevent the cacophony of intermixed signals in a
bidirectional architecture, mechanisms are needed to restrict information flow.
We explore deep recurrent neural net architectures in which bottom-up and
top-down signals are dynamically combined using attention. Modularity of the
architecture further restricts the sharing and communication of information.
Together, attention and modularity direct information flow, which leads to
reliable performance improvements in perceptual and language tasks, and in
particular improves robustness to distractions and noisy data. We demonstrate
on a variety of benchmarks in language modeling, sequential image
classification, video prediction and reinforcement learning that the
\emph{bidirectional} information flow can improve results over strong
baselines.
