  Reinforcement Learning (RL) has achieved great success in sequential
decision-making problems, but often at the cost of a large number of
agent-environment interactions. To improve sample efficiency, methods like
Reinforcement Learning from Expert Demonstrations (RLED) introduce external
expert demonstrations to facilitate agent exploration during the learning
process. In practice, these demonstrations, which are often collected from
human users, are costly and hence often constrained to a limited amount. How to
select the best set of human demonstrations that is most beneficial for
learning therefore becomes a major concern. This paper presents EARLY (Episodic
Active Learning from demonstration querY), an algorithm that enables a learning
agent to generate optimized queries of expert demonstrations in a
trajectory-based feature space. Based on a trajectory-level estimate of
uncertainty in the agent's current policy, EARLY determines the optimized
timing and content for feature-based queries. By querying episodic
demonstrations as opposed to isolated state-action pairs, EARLY improves the
human teaching experience and achieves better learning performance. We validate
the effectiveness of our method in three simulated navigation tasks of
increasing difficulty. The results show that our method is able to achieve
expert-level performance for all three tasks with convergence over 30\% faster
than other baseline methods when demonstrations are generated by simulated
oracle policies. The results of a follow-up pilot user study (N=18) further
validate that our method can still maintain a significantly better convergence
in the case of human expert demonstrators while achieving a better user
experience in perceived task load and consuming significantly less human time.
