  In order to successfully perform tasks specified by natural language
instructions, an artificial agent operating in a visual world needs to map
words, concepts, and actions from the instruction to visual elements in its
environment. This association is termed as Task-Oriented Grounding. In this
work, we propose a novel Dynamic Attention Network architecture for the
efficient multi-modal fusion of text and visual representations which can
generate a robust definition of state for the policy learner. Our model assumes
no prior knowledge from visual and textual domains and is an end to end
trainable. For a 3D visual world where the observation changes continuously,
the attention on the visual elements tends to be highly co-related from a
one-time step to the next. We term this as "Dynamic Attention". In this work,
we show that Dynamic Attention helps in achieving grounding and also aids in
the policy learning objective. Since most practical robotic applications take
place in the real world where the observation space is continuous, our
framework can be used as a generalized multi-modal fusion unit for robotic
control through natural language. We show the effectiveness of using 1D
convolution over Gated Attention Hadamard product on the rate of convergence of
the network. We demonstrate that the cell-state of a Long Short Term Memory
(LSTM) is a natural choice for modeling Dynamic Attention and shows through
visualization that the generated attention is very close to how humans tend to
focus on the environment.
