  Natural language is one of the most intuitive ways to express human intent.
However, translating instructions and commands towards robotic motion
generation and deployment in the real world is far from being an easy task. The
challenge of combining a robot's inherent low-level geometric and kinodynamic
constraints with a human's high-level semantic instructions traditionally is
solved using task-specific solutions with little generalizability between
hardware platforms, often with the use of static sets of target actions and
commands. This work instead proposes a flexible language-based framework that
allows a user to modify generic robotic trajectories. Our method leverages
pre-trained language models (BERT and CLIP) to encode the user's intent and
target objects directly from a free-form text input and scene images, fuses
geometrical features generated by a transformer encoder network, and finally
outputs trajectories using a transformer decoder, without the need of priors
related to the task or robot information. We significantly extend our own
previous work presented in Bucker et al. by expanding the trajectory
parametrization space to 3D and velocity as opposed to just XY movements. In
addition, we now train the model to use actual images of the objects in the
scene for context (as opposed to textual descriptions), and we evaluate the
system in a diverse set of scenarios beyond manipulation, such as aerial and
legged robots. Our simulated and real-life experiments demonstrate that our
transformer model can successfully follow human intent, modifying the shape and
speed of trajectories within multiple environments. Codebase available at:
https://github.com/arthurfenderbucker/LaTTe-Language-Trajectory-TransformEr.git
