  Many real-world continuous control problems are in the dilemma of weighing
the pros and cons, multi-objective reinforcement learning (MORL) serves as a
generic framework of learning control policies for different preferences over
objectives. However, the existing MORL methods either rely on multiple passes
of explicit search for finding the Pareto front and therefore are not
sample-efficient, or utilizes a shared policy network for coarse knowledge
sharing among policies. To boost the sample efficiency of MORL, we propose
Q-Pensieve, a policy improvement scheme that stores a collection of Q-snapshots
to jointly determine the policy update direction and thereby enables data
sharing at the policy level. We show that Q-Pensieve can be naturally
integrated with soft policy iteration with convergence guarantee. To
substantiate this concept, we propose the technique of Q replay buffer, which
stores the learned Q-networks from the past iterations, and arrive at a
practical actor-critic implementation. Through extensive experiments and an
ablation study, we demonstrate that with much fewer samples, the proposed
algorithm can outperform the benchmark MORL methods on a variety of MORL
benchmark tasks.
