  Recent research on structured exploration placed emphasis on identifying
novel states in the state space and incentivizing the agent to revisit them
through intrinsic reward bonuses. In this study, we question whether the
performance boost demonstrated through these methods is indeed due to the
discovery of structure in exploratory schedule of the agent or is the benefit
largely attributed to the perturbations in the policy and reward space
manifested in pursuit of structured exploration. In this study we investigate
the effect of perturbations in policy and reward spaces on the exploratory
behavior of the agent. We proceed to show that simple acts of perturbing the
policy just before the softmax layer and introduction of sporadic reward
bonuses into the domain can greatly enhance exploration in several domains of
the arcade learning environment. In light of these findings, we recommend
benchmarking any enhancements to structured exploration research against the
backdrop of noisy exploration.
