  The Visual-and-Language Navigation (VLN) task requires understanding a
textual instruction to navigate a natural indoor environment using only visual
information. While this is a trivial task for most humans, it is still an open
problem for AI models. In this work, we hypothesize that poor use of the visual
information available is at the core of the low performance of current models.
To support this hypothesis, we provide experimental evidence showing that
state-of-the-art models are not severely affected when they receive just
limited or even no visual data, indicating a strong overfitting to the textual
instructions. To encourage a more suitable use of the visual information, we
propose a new data augmentation method that fosters the inclusion of more
explicit visual information in the generation of textual navigational
instructions. Our main intuition is that current VLN datasets include textual
instructions that are intended to inform an expert navigator, such as a human,
but not a beginner visual navigational agent, such as a randomly initialized DL
model. Specifically, to bridge the visual semantic gap of current VLN datasets,
we take advantage of metadata available for the Matterport3D dataset that,
among others, includes information about object labels that are present in the
scenes. Training a state-of-the-art model with the new set of instructions
increase its performance by 8% in terms of success rate on unseen environments,
demonstrating the advantages of the proposed data augmentation method.
