  Game theoretic views of convention generally rest on notions of common
knowledge and hyper-rational models of individual behavior. However, decades of
work in behavioral economics have questioned the validity of both foundations.
Meanwhile, computational neuroscience has contributed a modernized 'dual
process' account of decision-making where model-free (MF) reinforcement
learning trades off with model-based (MB) reinforcement learning. The former
captures habitual and procedural learning while the latter captures choices
taken via explicit planning and deduction. Some conventions (e.g. international
treaties) are likely supported by cognition that resonates with the game
theoretic and MB accounts. However, convention formation may also occur via MF
mechanisms like habit learning; though this possibility has been understudied.
Here, we demonstrate that complex, large-scale conventions can emerge from MF
learning mechanisms. This suggests that some conventions may be supported by
habit-like cognition rather than explicit reasoning. We apply MF multi-agent
reinforcement learning to a temporo-spatially extended game with incomplete
information. In this game, large parts of the state space are reachable only by
collective action. However, heterogeneity of tastes makes such coordinated
action difficult: multiple equilibria are desirable for all players, but
subgroups prefer a particular equilibrium over all others. This creates a
coordination problem that can be solved by establishing a convention. We
investigate start-up and free rider subproblems as well as the effects of group
size, intensity of intrinsic preference, and salience on the emergence dynamics
of coordination conventions. Results of our simulations show agents establish
and switch between conventions, even working against their own preferred
outcome when doing so is necessary for effective coordination.
