  Generating music with emotion is an important task in automatic music
generation, in which emotion is evoked through a variety of musical elements
(such as pitch and duration) that change over time and collaborate with each
other. However, prior research on deep learning-based emotional music
generation has rarely explored the contribution of different musical elements
to emotions, let alone the deliberate manipulation of these elements to alter
the emotion of music, which is not conducive to fine-grained element-level
control over emotions. To address this gap, we present a novel approach
employing musical element-based regularization in the latent space to
disentangle distinct elements, investigate their roles in distinguishing
emotions, and further manipulate elements to alter musical emotions.
Specifically, we propose a novel VQ-VAE-based model named MusER. MusER
incorporates a regularization loss to enforce the correspondence between the
musical element sequences and the specific dimensions of latent variable
sequences, providing a new solution for disentangling discrete sequences.
Taking advantage of the disentangled latent vectors, a two-level decoding
strategy that includes multiple decoders attending to latent vectors with
different semantics is devised to better predict the elements. By visualizing
latent space, we conclude that MusER yields a disentangled and interpretable
latent space and gain insights into the contribution of distinct elements to
the emotional dimensions (i.e., arousal and valence). Experimental results
demonstrate that MusER outperforms the state-of-the-art models for generating
emotional music in both objective and subjective evaluation. Besides, we
rearrange music through element transfer and attempt to alter the emotion of
music by transferring emotion-distinguishable elements.
