  Human feedback is widely used to train agents in many domains. However,
previous works rarely consider the uncertainty when humans provide feedback,
especially in cases that the optimal actions are not obvious to the trainers.
For example, the reward of a sub-optimal action can be stochastic and sometimes
exceeds that of the optimal action, which is common in games or real-world.
Trainers are likely to provide positive feedback to sub-optimal actions,
negative feedback to the optimal actions and even do not provide feedback in
some confusing situations. Existing works, which utilize the Expectation
Maximization (EM) algorithm and treat the feedback model as hidden parameters,
do not consider uncertainties in the learning environment and human feedback.
To address this challenge, we introduce a novel feedback model that considers
the uncertainty of human feedback. However, this incurs intractable calculus in
the EM algorithm. To this end, we propose a novel approximate EM algorithm, in
which we approximate the expectation step with the Gradient Descent method.
Experimental results in both synthetic scenarios and two real-world scenarios
with human participants demonstrate the superior performance of our proposed
approach.
