  Modern reinforcement learning (RL) struggles to capture real-world
cause-and-effect dynamics, leading to inefficient exploration due to extensive
trial-and-error actions. While recent efforts to improve agent exploration have
leveraged causal discovery, they often make unrealistic assumptions of causal
variables in the environments. In this paper, we introduce a novel framework,
Variable-Agnostic Causal Exploration for Reinforcement Learning (VACERL),
incorporating causal relationships to drive exploration in RL without
specifying environmental causal variables. Our approach automatically
identifies crucial observation-action steps associated with key variables using
attention mechanisms. Subsequently, it constructs the causal graph connecting
these steps, which guides the agent towards observation-action pairs with
greater causal influence on task completion. This can be leveraged to generate
intrinsic rewards or establish a hierarchy of subgoals to enhance exploration
efficiency. Experimental results showcase a significant improvement in agent
performance in grid-world, 2d games and robotic domains, particularly in
scenarios with sparse rewards and noisy actions, such as the notorious Noisy-TV
environments.
