  In this work, we focus on the problem of grounding language by training an
agent to follow a set of natural language instructions and navigate to a target
object in an environment. The agent receives visual information through raw
pixels and a natural language instruction telling what task needs to be
achieved and is trained in an end-to-end way. We develop an attention mechanism
for multi-modal fusion of visual and textual modalities that allows the agent
to learn to complete the task and achieve language grounding. Our experimental
results show that our attention mechanism outperforms the existing multi-modal
fusion mechanisms proposed for both 2D and 3D environments in order to solve
the above-mentioned task in terms of both speed and success rate. We show that
the learnt textual representations are semantically meaningful as they follow
vector arithmetic in the embedding space. The effectiveness of our attention
approach over the contemporary fusion mechanisms is also highlighted from the
textual embeddings learnt by the different approaches. We also show that our
model generalizes effectively to unseen scenarios and exhibit zero-shot
generalization capabilities both in 2D and 3D environments. The code for our 2D
environment as well as the models that we developed for both 2D and 3D are
available at https://github.com/rl-lang-grounding/rl-lang-ground.
