  In this work, we focus on multi-step manipulation tasks that involve
long-horizon planning and considers progress reversal. Such tasks interlace
high-level reasoning that consists of the expected states that can be attained
to achieve an overall task and low-level reasoning that decides what actions
will yield these states. We propose a sample efficient Previous Action
Conditioned Robotic Manipulation Network (PAC-RoManNet) to learn the
action-value functions and predict manipulation action candidates from visual
observation of the scene and action-value predictions of the previous action.
We define a Task Progress based Gaussian (TPG) reward function that computes
the reward based on actions that lead to successful motion primitives and
progress towards the overall task goal. To balance the ratio of
exploration/exploitation, we introduce a Loss Adjusted Exploration (LAE) policy
that determines actions from the action candidates according to the Boltzmann
distribution of loss estimates. We demonstrate the effectiveness of our
approach by training PAC-RoManNet to learn several challenging multi-step
robotic manipulation tasks in both simulation and real-world. Experimental
results show that our method outperforms the existing methods and achieves
state-of-the-art performance in terms of success rate and action efficiency.
The ablation studies show that TPG and LAE are especially beneficial for tasks
like multiple block stacking. Additional experiments on Ravens-10 benchmark
tasks suggest good generalizability of the proposed PAC-RoManNet.
