  This paper presents a deep reinforcement learning algorithm for online
accompaniment generation, with potential for real-time interactive
human-machine duet improvisation. Different from offline music generation and
harmonization, online music accompaniment requires the algorithm to respond to
human input and generate the machine counterpart in a sequential order. We cast
this as a reinforcement learning problem, where the generation agent learns a
policy to generate a musical note (action) based on previously generated
context (state). The key of this algorithm is the well-functioning reward
model. Instead of defining it using music composition rules, we learn this
model from monophonic and polyphonic training data. This model considers the
compatibility of the machine-generated note with both the machine-generated
context and the human-generated context. Experiments show that this algorithm
is able to respond to the human part and generate a melodic, harmonic and
diverse machine part. Subjective evaluations on preferences show that the
proposed algorithm generates music pieces of higher quality than the baseline
method.
