  Although many reinforcement learning methods have been proposed for learning
the optimal solutions in single-agent continuous-action domains, multiagent
coordination domains with continuous actions have received relatively few
investigations. In this paper, we propose an independent learner hierarchical
method, named Sample Continuous Coordination with recursive Frequency Maximum
Q-Value (SCC-rFMQ), which divides the cooperative problem with continuous
actions into two layers. The first layer samples a finite set of actions from
the continuous action spaces by a re-sampling mechanism with variable
exploratory rates, and the second layer evaluates the actions in the sampled
action set and updates the policy using a reinforcement learning cooperative
method. By constructing cooperative mechanisms at both levels, SCC-rFMQ can
handle cooperative problems in continuous action cooperative Markov games
effectively. The effectiveness of SCC-rFMQ is experimentally demonstrated on
two well-designed games, i.e., a continuous version of the climbing game and a
cooperative version of the boat problem. Experimental results show that
SCC-rFMQ outperforms other reinforcement learning algorithms.
