  We present a new class of structured reinforcement learning
policy-architectures, Implicit Two-Tower (ITT) policies, where the actions are
chosen based on the attention scores of their learnable latent representations
with those of the input states. By explicitly disentangling action from state
processing in the policy stack, we achieve two main goals: substantial
computational gains and better performance. Our architectures are compatible
with both: discrete and continuous action spaces. By conducting tests on 15
environments from OpenAI Gym and DeepMind Control Suite, we show that
ITT-architectures are particularly suited for blackbox/evolutionary
optimization and the corresponding policy training algorithms outperform their
vanilla unstructured implicit counterparts as well as commonly used explicit
policies. We complement our analysis by showing how techniques such as hashing
and lazy tower updates, critically relying on the two-tower structure of ITTs,
can be applied to obtain additional computational improvements.
