  In situations where explicit communication is limited, human collaborators
act by learning to: (i) infer meaning behind their partner's actions, and (ii)
convey private information about the state to their partner implicitly through
actions. The first component of this learning process has been well-studied in
multi-agent systems, whereas the second --- which is equally crucial for
successful collaboration --- has not. To mimic both components mentioned above,
thereby completing the learning process, we introduce a novel algorithm: Policy
Belief Learning (PBL). PBL uses a belief module to model the other agent's
private information and a policy module to form a distribution over actions
informed by the belief module. Furthermore, to encourage communication by
actions, we propose a novel auxiliary reward which incentivizes one agent to
help its partner to make correct inferences about its private information. The
auxiliary reward for communication is integrated into the learning of the
policy module. We evaluate our approach on a set of environments including a
matrix game, particle environment and the non-competitive bidding problem from
contract bridge. We show empirically that this auxiliary reward is effective
and easy to generalize. These results demonstrate that our PBL algorithm can
produce strong pairs of agents in collaborative games where explicit
communication is disabled.
