  Standard reinforcement learning (RL) algorithms assume that the observation
of the next state comes instantaneously and at no cost. In a wide variety of
sequential decision making tasks ranging from medical treatment to scientific
discovery, however, multiple classes of state observations are possible, each
of which has an associated cost. We propose the active measure RL framework
(Amrl) as an initial solution to this problem where the agent learns to
maximize the costed return, which we define as the discounted sum of rewards
minus the sum of observation costs. Our empirical evaluation demonstrates that
Amrl-Q agents are able to learn a policy and state estimator in parallel during
online training. During training the agent naturally shifts from its reliance
on costly measurements of the environment to its state estimator in order to
increase its reward. It does this without harm to the learned policy. Our
results show that the Amrl-Q agent learns at a rate similar to standard
Q-learning and Dyna-Q. Critically, by utilizing an active strategy, Amrl-Q
achieves a higher costed return.
