  Successful negotiators must learn how to balance optimizing for self-interest
and cooperation. Yet current artificial negotiation agents often heavily depend
on the quality of the static datasets they were trained on, limiting their
capacity to fashion an adaptive response balancing self-interest and
cooperation. For this reason, we find that these agents can achieve either high
utility or cooperation, but not both. To address this, we introduce a targeted
data acquisition framework where we guide the exploration of a reinforcement
learning agent using annotations from an expert oracle. The guided exploration
incentivizes the learning agent to go beyond its static dataset and develop new
negotiation strategies. We show that this enables our agents to obtain
higher-reward and more Pareto-optimal solutions when negotiating with both
simulated and human partners compared to standard supervised learning and
reinforcement learning methods. This trend additionally holds when comparing
agents using our targeted data acquisition framework to variants of agents
trained with a mix of supervised learning and reinforcement learning, or to
agents using tailored reward functions that explicitly optimize for utility and
Pareto-optimality.
