  We present Piano Genie, an intelligent controller which allows non-musicians
to improvise on the piano. With Piano Genie, a user performs on a simple
interface with eight buttons, and their performance is decoded into the space
of plausible piano music in real time. To learn a suitable mapping procedure
for this problem, we train recurrent neural network autoencoders with discrete
bottlenecks: an encoder learns an appropriate sequence of buttons corresponding
to a piano piece, and a decoder learns to map this sequence back to the
original piece. During performance, we substitute a user's input for the
encoder output, and play the decoder's prediction each time the user presses a
button. To improve the intuitiveness of Piano Genie's performance behavior, we
impose musically meaningful constraints over the encoder's outputs.
