Algorithmic Songwriting with ALYSIA
Margareta Ackerman
David Loker
This paper introduces ALYSIA: Automated LYrical SongwrIting Application. ALYSIA is based on a machine learning model using Random Forests, and we discuss its success at pitch and rhythm prediction. Next, we show how ALYSIA was used to create original pop songs that were subsequently recorded and produced. Finally, we discuss our vision for the future of Automated Songwriting for both co-creative and autonomous systems.
main.tex
\documentclass[runningheads,a4paper]{llncs}
\usepackage[english]{babel}
\usepackage[utf8x]{inputenc}
%\usepackage{aaai}
\usepackage{graphicx}
\usepackage{url}
\usepackage{hyperref}
\usepackage{caption, comment}
%\usepackage{fullpage}


\title{Algorithmic Songwriting with ALYSIA}

\author{}
\institute{}

%\begin{comment}
\author{Margareta Ackerman \and David Loker}

\institute{Department of Computer Science, San Jose State University\\
San Jose, CA USA\\
\email{margareta.ackerman@sjsu.edu}\\
\and
Orbitwerks\\
San Jose, CA  USA\\
\email{dloker@gmail.com}}
%\end{comment}

\begin{document}
\maketitle



\begin{abstract}
This paper introduces ALYSIA: Automated LYrical SongwrIting Application.  ALYSIA is based on a machine learning model using Random Forests, and we discuss its success at pitch and rhythm prediction. Next, we show how ALYSIA was used to create original pop songs that were subsequently recorded and produced. Finally, we discuss our vision for the future of Automated Songwriting for both co-creative and autonomous systems. 
\end{abstract}

\section{Introduction}

Music production has traditionally relied on a wide range of expertise far exceeding the capabilities of most individuals. Recent advancements in music technology and the parallel development of electronic music has brought the joy of music-making to the masses. Among the countless amateurs who enjoy expressing their creativity through music-making, there has emerged a newly-leveled playing field for professional and semi-professional musicians who are able to independently produce music with no more than a personal computer. 

Technological advances have greatly simplified music production through systems such as Logic Pro\footnote{http://www.apple.com/logic-pro/}, Live\footnote{https://www.ableton.com/en/live/}, or even the freely available Garage Band\footnote{http://www.apple.com/mac/garageband/}. In addition to placing a wide range of instruments at our fingertips, without requiring the corresponding training, these systems provide a variety of mastering tools that enable the creation of high quality music with relative ease. Yet, despite this paradigm shift, the writing of songs - comprised of both melody and lyrics - remains an exclusive domain. A careful examination of electronic music reveals these limitations. The genre relies heavily on remixing existing music and much attention is paid to sound design, which does not require original composition or its juxtaposition with lyrics. 



The creation of a new song continues to rely on diverse skills rarely possessed by one individual. The score alone calls for a combination of poetic ability to create the lyrics, musical composition for  the melody, and expertise in combining lyrics and melody into a coherent, meaningful and aesthetically pleasing whole. Once the score is complete, vocal skill and production expertise are needed to turn it into a finalized musical piece. While some may be satisfied with remixing existing music, the creation of truly original songs is often desired, in part due to restrictions associated with performing and remaking existing works. Can technology assist in the creation of original songs, allowing users to express their creativity through the skills they possess while letting technology handle the rest? 

We introduce ALYSIA, a songwriting system specifically designed to help both musicians and amateurs write  songs, with the goal of producing professional quality music. ALYSIA focuses on one of the most challenging aspects of the songwriting process, enabling the human creator to discover novel melodies to accompany lyrics. To the best of our knowledge, ALYSIA is the first songwriting system to utilize formal machine learning methodology. This lets us evaluate the success of our model based on its ability to correctly predict the pitch and rhythm of notes in the test set (which was not used to train the model).  On a corpus of pop music, the system achieves an impressive accuracy of 86.79\% on rhythms and 72.28\% on scale-degrees. 

%It has been argued that any melody may sound acceptable when combined with a suitably chosen harmonic structure~\cite{papadopoulos1999ai}. This may be relevant when wishing to make pleasing music, but our goal here is more complex: We wish to discover melodies that \emph{work well with the underlying lyrics, and conform with a given music style.}  

%For example, the rhythm of the melody needs to adhere to the syllable stress of the lyrics~\footnote{Consider for example the word ``rainbow.'' If ``bows'' is emphasized by having a much longer notes than ``rain,'' the melody may be unfit for the lyrics, even if the melody is otherwise pleasing.}. Adhering to a specific genre further complicates this problem, as, for example, consecutive repeated notes occur much more frequently in pop than in opera, and the range varies notably between these genres. Composers and songwriters are, consciously or unconsciously, aware of the many different constraints that help them to effectively explore the search space of possible melodies. Our goal here is to automatically do the same, but letting the system discover the rules, making it genre independent. 

%Similar to the ground-breaking work of David Cope~\cite{cope1996experiments}, 

%ALYSIA was originally conceived to fill the need of its makers. One of the authors, Margareta Ackerman, had a long standing dream to write and produce original songs. ALYSIA enables the creation of songs in any given genre without the need for  songwriting expertise. Using ALYSIA, Margareta was able to write pop music for the first time.

%, despite having no expertise or experience with songwriting. 

%Underlying ALYSIA is a machine learning model. 




%In addition, our model has no explicitly built-in musical knowledge, letting it automatically determine which features are important based on the corpus, giving ALYSIA true genre flexibility. 

%Our contributions are twofold. First, ALYSIA is based on a machine learning model and has no rule-based components, which allows it to easily adapt to different musical styles. Leveraging machine learning methodology, we are able to evaluate the success of our model, as well as determine which features are most important for melody generation. Second, we present an approach to computer-assisted songwriting, bringing the creative experience of songwriting to those who otherwise lack this ability. 

%Its prediction accuracy provides a source of automated evaluation, ensuring that the user selects among reasonable melody options. 

 In the generation phase, ALYSIA is given short lyrical phrases, for which it can then provide numerous melody options in the style of the given corpus. As it stands today, ALYSIA is a co-creative system that allows the user to retain a sense of creative control while eliminating one of the most essential skills traditionally required for songwriting: the ability to effectively explore the space of potential accompanying melodies. As a co-creative songwriting partner, ALYSIA assists human creators in the song-making progress by filling in gaps in the user's expertise. 
 
%Novices lack the ability to thoroughly explore a space and are less creative than experts~\cite{sawyer2013zig}. As such, those who are inexperienced in composition and songwriting are unable to effectively explore the space of possible melodies. By allowing a user to quickly and easily explore the space of potential accompaniments for a given lyrical line, ALYSIA makes songwriting accessible to those who could not previously engage in this creative process. 

Helping people be creative, particularly in a manner that aids their social engagement both on and offline, can have substantial positive impact on their sense of well-being and happiness~\cite{gauntlett2013making}. Technological advancement has led to the radical expansion of those who are able to engage in music making, and made possible today's large online  communities for sharing musical works. Widening access to original songwriting stands not only to ease the path to professional musicianship, but also makes it possible for more people to enjoy the psychological benefits of engaging in this creative and socially rewarding activity.

Without any changes to the system, ALYSIA could be used to create complete melodies without human interference by simply selecting her first suggestion for each line of lyrics. However, a little human interaction can result in better music and gives the user creative freedom that would be lacking from even the best autonomous system. 

%- which we find to be the novel and impactful contribution of our work

%However, using the system in a co-creative environment with relatively little work on the part of the user 

%; yet, we stress that used as co-creative system, ALYSIA can be used to create high quality music. 

%As the system continues to grow, it will be expanded into a more helpful co-creative system as well as progress towards becoming an autonomous creative agent.

%Our contributions are twofold. First, ALYSIA is based on a machine learning model and has no rule-based components, which allows it to easily adapt to different musical styles. Leveraging machine learning methodology, we are able to evaluate the success of our model, as well as determine which features are most important for melody generation. Second, we present an approach to computer-assisted songwriting, bringing the creative experience of songwriting to those who otherwise lack this ability. 

%This allows both professional and amateur musicians to create original songs while focusing on the aspects of the process that best fit their interests.  

%Our approach to co-creative songwriting may have independent interest.

%and may be incorporated into other systems. 

We begin with an examination of previous work and how it relates to our contributions, followed by a technical description of our system. We then move onto the application of ALYSIA to songwriting, starting with a detailed discussion of our proposed co-creative process. Finally, we showcase three musical pieces created with ALYSIA, and conclude with a discussion of our vision for the future of algorithmic songwriting. 

%starting with two pop songs, one of which has been recorded and produced. 

%Lastly, we show how ALYSIA can be used for musical genre translation with a conversion of a 1917 Vaudeville song \emph{I'm Always Chasing Rainbows} into the modern pop style.

%The music is credited to Harry Carroll, although the melody was in fact adapted from Frédéric Chopin's Fantaisie-Impromptu. Lyrics were written by  Joseph McCarthy. The songs appearance in Broadway musicals and motion pictures contributed to its popularity. As it was published in 1917, ``I'm always chasing rainbows'' is in public domain. 



%\newpage
\section{Previous work}

Algorithmic composition has a rich history dating back to the 1950s. A wide range of techniques have been applied to music generation (with no lyrics), spanning from expert systems to neural networks.  See a survey by Fern{\'a}ndez and Vico~\cite{fernandez2013ai} for an excellent overview. 

Our focus here is on the creation of songs, which introduces a lyrical component to algorithmic composition. Algorithmic songwriting is relatively new, and so far the state of art addresses it using Markov chains. For example, M.U. Sucus-Apparatusf~\cite{toivanen2013automatical} by Toivanen et al, is a system that was used for the generation of Finnish art songs. Rhythm patterns are randomly chosen from among those typically found in art songs. Chord progressions are  subsequently generated by using second order Markov chains. Lastly, pitches are generated using a joint probability distribution based on chords and the previous note. M.U. Sucus-Apparatusf integrates the entire songwriting process, from lyric generation to melody and accompaniment. Yet, as with most previous systems, the major weakness is a lack of clear phrase structure \cite{toivanen2013automatical}. 

%Lastly, pitches are generated using the probability distribution that is a product of the probability distribution determined by the underlying chord and the distribution given by the previous melody note (where small intervals between consecutive notes are more likely).

Another full-cycle automated songwriting system comes from the work of Scirea et al~\cite{scirea2015smug}. Their songwriting system SMUG was used to write songs from academic papers. The methodology relies on the evolution of Markov chains. It uses a corpus of mixed-genre popular songs, and integrates several rules, such as the high probability of rests at the ends of words. 

Monteith at al. \cite{monteith2012automatic} study the generation of melodic accompaniments. While utilizing a corpus, the system works by generating hundreds of corpus-driven random options guided by a few rules, and then chooses among them with an evaluation criteria that incorporates some musical knowledge. This process is applied for both the rhythm and melody generation. For example, for generating rhythms, the system produces 100 possible downbeat assignments. The text of each line is distributed across four measures, so four syllables are randomly selected to carry a downbeat. 

In other related works, Nichols~\cite{nichols2009lyric} studies a sub-component of the problem we address here, particularly lyric-based rhythm suggestion. He considers the set of all accompanying rhythms, and defines a fully rule-based function to evaluate them, via considerations such as rare word emphasis through strong beats. As future work, Nichols suggests solving the rhythm generation processes through machine learning techniques, which is one of our contributions here. Lastly,  in the work of Oliveira~\cite{oliveira2015tra}, the pipeline is inverted and lyrics are generated based on the rhythm of the provided melody. 

Our approach does not encode any musical rules and uses Random forests instead of Markov chains. The lack of directly encoded musical knowledge gives our system true genre independence. Musical genres are incredibly diverse, and rules that make sense for one genre can make it impossible to generate songs in another. For example, penalizing melodies with many repeating consecutive pitches is often a sensible choice, but it is unsuited to pop music, where such sequences are common. A  machine learning model that does not explicitly encode musical expertise not only applies to existing genres, but can also adapt to all future musical genres, which can differ from prior music in unpredictable ways. 



%\cite{monteith2012automatic} study the generation of melodic accompaniments. While utilizing a corpus, the system works by generating hundreds of corpus-driven random options guided by a few rules, and then chooses among them with an evaluation criteria that incorporates some musical knowledge. This process is applied for both the rhythm and melody generation. More specifically, for generating rhythms, the system produces 100 possible downbeat assignments. The text of each line is distributed across four measures, so four syllables are randomly selected to carry a downbeat. To choose among the 100 rhythmic patterns, a rule-based evaluation criterion is applied which prefers downbeat assignments that fall on stressed syllables and at the start of words, as well as preferring to space syllables more evenly across the four measures. To generate the melodies, an n-gram with $n=4$ is created based on the corpus. Each musical line is initialized with the same three randomly generated notes. Then a rule-based criteria selects among these, for example, by assigning higher scores to melodies with a greater number of distinct pitches, fewer repetitions, and tighter range. 


Random forests offer some advantages over using Markov chains. One draw-back of Markov chains is that while they can model likelihoods of sequences, they do not encapsulate the ``why'' of one note being chosen over another after the same initial sequence. For example, if note A is the first note, why is the next note B most of the time? Perhaps it is only B most of the time if the key signature is C-major, and it is actually F\# most of the time under a different context. Such context can be discovered and used in systems like random forests given a suitable feature set. Additionally, we rely on machine learning methodology to formally evaluate our model, which has not been done in previous work on automated songwriting.  Another difference is that we rely on a co-creative process, which aids with our vision to create high quality music. Finally, ALYSIA is the first songwriting system whose songs were recorded and produced.



%Despite the use of data in previous work, ALYSIA is the first songwriting system that uses machine learning models and incorporates a formal evaluation of these models prior to generation. Ours is also the first system that does not directly encode any musical knowledge, allowing the system to automatically model the structure of the music in the corpus. This gives our system a new level of flexibility and responsiveness to the corpus, opening the door to automated genre translation. Lastly, to the best of our knowledge, ours is the first system that has been used to create production-quality songs that were subsequently recorded and produced. 


\begin{comment}
By the spring of 2016, ALYSIA had been developed, three pop songs written using the system, and a draft of this paper was complete. The songs and a brief description of the system have been posted since August 2016 (link omitted for double-blind review). In late September 2016, SONY released a pair of pop songs written using their Flow Machine system\footnote{\begin{scriptsize}\url{http://www.digitalmusicnews.com/2016/09/28/sony-pop-songs-artificial-intelligence/}{}\end{scriptsize}}, claiming to be the first pop songs made using artificial intelligence.  Our work has been complete and posted online before SONY released their songs, and the research was done independently. Furthermore, we were unable to locate the scientific work related to the creation of these songs\footnote{SONY has published how Flow Machines were used to generated music without lyrics (see, for example, ~\cite{papadopoulos2016assisted} and \cite{pachet2016joyful}). In previous published works, Flow Machine were used to write lead sheets through a combination of Markov sequences and stochastic temporal constraints~\cite{papadopoulos2016assisted}. This suggests that the methodology used for SONY's pop songs is likely entirely different from ours. }. As such, the current paper is, to our best knowledge, the first scientific paper about the creation of recorded pop songs using artificial intelligence.
\end{comment}




\section{The making of ALYSIA}

ALYSIA is an entirely data-driven system that is based upon a prediction model. We chose to construct two models. The first predicts note duration, which we refer to as the \emph{rhythm model}. The second predicts the scale degree of a note, with possible accidentals, which we refer to as the \emph{melody model}. Both models rely on a similar set of features, though depending on which model is run first, one model will have access to extra information. In our case, the melody model benefits from the predictions made by the rhythm model. Note that combing the models would increase the number of output classes, requiring a larger corpus. 

%For example, if we predict note duration first, we may use the duration as part of the model for predicting the melody.

The most important component of our model is the set of features. The aim is to construct features that will allow the model to learn the mechanics and technicalities behind the structure of songs and their composition. 

%As with any machine learning model, there may be additional important features, which we will continue to explore in future work. 

Since we provide melodies for text provided by the user, our training set consists of vocal lines of music, and not arbitrary melodies. The corpus consists of Music-XML (MXL) files, with a single instrument corresponding to the vocal line and the accompanying lyrics. As such, each note has a corresponding syllable. 

%MORE ABOUT DATA?>?!??! MXL FILES, COMBINATION OF NOTES AND LYRICS

The system consists of five distinct components.
\begin{enumerate}
\item The corpus
\item Feature extraction 
\item Model building and evaluation
\item User lyrics to features
\item Song generation
%\item MIDI and MXL Translator
\end{enumerate}

We now discuss each of these in detail. 


\subsection{Feature Extraction}

In order to build a model, we first extract a set of features from the Music-XML (MXL) files. 

%The MXL files contain notes from several pop songs. We restrict our attention to the vocal parts.

For each note, we extract the following features:
\begin{itemize}
\item First Measure - A boolean variable indicating whether or not the current note belongs to the first measure of the piece
\item Key Signature - The key signature that applies to the current note
\item Time Signature - The time signature that applies to the current note
\item Offset - The number of beats since the start of the music
\item Offset within Measure - The number of beats since the start of the measure
\item Duration - The length of the note
\item Scale Degree - The scale degree of the note (1-7)
\item Accidental - The accidental of the note (flat, sharp, or none)
\item Beat Strength - The strength of beat as defined by music21. We include the continuous and categorical version (Beat Strength Factor) of this variable. 
\item Offbeat - A boolean variable specifying whether or not the note is offbeat
\item Syllable Type\textbf{*} - Classifies the current syllable in one of the following four categories: Single (the word consists of a single syllable), Begin (the current syllable is the first one in its word), Middle (the current syllable occurs in the middle of its word), End  (the current syllable is the last one in its word).
\item Syllable Number\textbf{*} - The syllable number within its word
\item Word Frequency\textbf{*} - The word frequency of the word which the current note/syllable is part of. This value is obtained through the dictionary frequency as calculated by Pythons NLTK (\href{http://www.nltk.org/}{http://www.nltk.org/}) library using the Brown corpus~\footnote{Brown corpus: \href{http://www.hit.uib.no/icame/brown/bcm.html}{http://www.hit.uib.no/icame/brown/bcm.html}}. 
\item Word Rarity\textbf{*} - A function of word frequency, as defined by \cite{nichols2009lyric}. $WordRarity = 2(1-\frac{\log_{10}(WordFrequency)}{7}).$
\item Word Vowel Strength\textbf{*} - The word vowel strength (primary, secondary, or none) as indicated by the  CMUDict v0.07.\footnote{CMUDict v0.07, Carnegie Mellon University.}
%http://www.speech.cs.cmu.edu/cgi-bin/cmudict
%\footnote{Copyright (C) 1993-2015 Carnegie Mellon University. All rights reserved.}
\item Number of Vowels in the Word\textbf{*} - The number of vowels found in the word corresponding to the current note/syllable. 
\item Scale Degree, Accidental, and Duration of previous 5 notes
\end{itemize}

The featured marked with an * are generated from lyrics found within the vocal line. These same features must be generated using the user's lyrics when ALYSIA is used in generation mode. The features marked with an * were used by \cite{nichols2009lyric} as part of a rule-based evaluation criteria for rhythm suggestion. 

%Word frequencies and rarities are done relative to the standard python Brown corpus. 

\subsection{Model Building and Evaluation}

Using R, we train two models using random forests. Random forests were chosen for several reasons. They are well-suited for large numbers of categorical variables. Additionally, they allow non-linearity in combining features, without an explosion in the required size of the training set, in order to avoid overfitting.  Non-linearity makes random forests more powerful than linear models such as logistic regression. We randomly split the data using stratified sampling into a training set (75\% of the data) with the rest for testing.   

%\subsection{Rhythm Model}
The accuracy of the rhythm model on test data was 86.79\%. Figure~\ref{fig:confRhythm} shows the confusion matrix. The row labels correspond to the duration found in the test sets, and column labels are the predicted labels. The number after the underscore corresponds to the number of dots in the durations. So, for example, a dotted sixteenth note is 16th\_1. 

As you can see in the confusion matrix, the model tends to perform better on rhythms which occur more frequently in the data set. A larger data set will enable us to test whether the trend will continue. 

\begin{figure*}
    \advance\leftskip-2cm
    \includegraphics[width = 16cm]{rhythmModel.jpg}
    \caption{Confusion matrix for Rhythm Model. Cell ($i$,$j$) in the above table specifies how often a node with duration type $i$ was classified as having during $j$. The last column shows the error rate for each duration type (NaN indicating that no notes of this duration were found in the corpus).}
    \label{fig:confRhythm}
\end{figure*}


%\subsection{Melody Model}
%\newpage

The accuracy of the melody model was 72.28\%. Figure~\ref{fig:confRhythm} shows the confusion matrix. The numbers correspond to the scale degrees, and the label after the underscore represents the accidentals. So, for example, 1\_none represents the first scale degree without accidentals. As with the rhythm model, we see that accuracy is higher when there are more examples. 

Lastly, our machine learning model allows us to evaluate the importance of each feature. Figure~\ref{fig:modelDecAccuracy} depicts the mean decrease Gini (which measures node impurity in tree-based classification) of each feature in the rhythm model (left) and the melody model (right). In both models, all 5 previous notes' scale degrees are some of the most important features. This suggests that including features for additional previous notes may be helpful. This also explains why Markov chains seem to do well for this task. However, we see that other features are also prominent. 

For example, the rhythm model is most heavily influenced by beat strength, revealing the potential of using a data-driven approach over Markov chains, which do not give access to such features. Finally, it is important to note that lyrical features such as word rarity and frequency also play an important role. 

\begin{figure*}
    \advance\leftskip-2cm
    \includegraphics[width = 16cm]{Confusion_melody}
    \caption{Confusion matrix for Melody Model. Cell ($i$,$j$) in the above table specifies how often scale degree $i$ was classified as scale degree $j$. The last column shows the error rate for each scale degree.}
    \label{fig:confRhythm}
\end{figure*}


\begin{figure*}
    \advance\leftskip-1cm
    \includegraphics[angle=90, height = 22.5cm]{rhythmMelodyGini}
    \caption{Importance plot for the rhythm model (left) and the melody model (right), representing the significance of each feature in the respective models.}
    \label{fig:modelDecAccuracy}
\end{figure*}


%\begin{figure*}
%    \advance\leftskip-1cm
%    \includegraphics[width = 18cm]{paramsMelody}
%    \caption{Melody importance plot}
%    \label{fig:confRhythm}
%\end{figure*}


%We randomly split the data using stratified sampling into a training set (75\% of the data) with the rest for testing. The accuracy of the rhythm model on test data was 0.8396, and the accuracy of the melody model was 0.7715. 

%In our case, we chose to allow the melody to have access to the rhythm, and during generation we generate rhythm followed by melody.

\subsection{User Lyrics to Features} In order to generate music for a new set of lyrics using our models, we need to extract the same set of lyric features we were using for our models. This was done using a python script, which outputs a feature matrix for importing into R.

\subsection{Song Generation} Using R, we loop through the lyric feature set one sentence at a time. For each line, we read the feature set from the lyrics given by the user and generate the rhythm, followed by the melody. Note that we generate one note at a time for each syllable in the lyrical phrases. We also keep track of 5 previous notes, since these are very important features in the model. Finally, we generate a file detailing the generated melodies for each line. There may be many depending on how many melodies per line were requested.



\subsection{Corpus}
Our corpus consists of 12,358 observations on 59 features that we extract from MXL files of 24 pop songs. From each song, only the melody lines and lyrics were used. For model building and evaluation, the data is split using stratified sampling along the outcome variable, which is scale degree with accent for the melody model, and note duration for the rhythm model. Seventy-five percent of the data is used for training, while the remaining data is used for evaluation.

%Combined, the MXL files contain 12,358 observations on 59 features that we extract. 
Please note that unlike MP3 files, MXL files containing lyrics are much more difficult to acquire, limiting the size of our data. Previous work on automated songwriting at times does not use any training (it is instead rule-based), or  trains separately on lyrics and melodies - which eliminates the possibility of learning how they interact. The size of the training set in previous work is often omitted\footnote{Due to the difficulties in attaining MXL files, we are currently creating a new larger corpus of songs, which will be analyzed in future work.}. 


\subsection{Parameters} 
When used in generation mode, ALYSIA allows the user to tune the following parameters:

%The final step in the generation phase of our process is to take what the models generate and translate the output to a MIDI file and MXL file. This is done via Python using the Music21 libraries. 
\subsubsection{Explore/Exploit} This parameter determines how heavily we lean on the model. Specifically, for each scale degree/note duration we generate, our model outputs a distribution over all possible outcomes. This parameter determines how many independent draws we make from this distribution. The final resulting note is the most common draw, with ties being broken by the original outcome distribution. That is, if scale degree 1 and 2 are tied after four draws, we take the scale degree that was originally more likely in the distribution output by the model for this data point. Given this definition, a higher explore/exploit parameter value means we exploit because we are almost always going to output the scale degree or duration that has the highest probability of occurring. This parameter allows us to favor the scale degrees and durations that are most likely, versus potentially taking a more varied approach and generating music that could be considered more experimental.

\subsubsection{Melody Count} We generate a set number of melodies per line, outputting them to a file for later translation into midi and MXL files.

\subsubsection{Rhythm Restriction} We allow the user to restrict the possible rhythmic outcome if they would like to omit certain note durations. For example, the user may want to disallow whole notes, or some faster notes such as a 32nd note.



\newpage
\section{The Co-Creative Process of Songwriting with ALYSIA}

In this section, we describe the co-creative process of songwriting with ALYSIA. This is a new approach to writing songs that requires minimal to no musical training. ALYSIA makes it easy to explore melodic lines, reducing songwriting to the ability to select melodies based on one's musical taste. 



The user provides ALYSIA with the lyrics broken down into separate lines. Then, the system gives the specified number of melodies (notes/rhythm combinations) to which the given lyrics can be sang. The number of melodies per line is specified by the user. ALYSIA outputs the melodies in both MXL and MIDI format. The MIDI form is particularly helpful, since it allows the user to play the variations, select the desired one, and directly utilize the MIDI to produce the song. 

We now describe the workflow we used with ALYSIA. We typically ask for between 15 to 30 melodic variations per line of text. Among these, we select between three and ten melodic lines. It should be noted that nearly all of ALYSIA's suggestions are reasonable. For example, Figure~\ref{fig:variations} lists the first 12 melody options provided by ALYSIA for the phrase \emph{everywhere I look I find you looking back.} All of these melodies  work fairly well with the underlying text and could be extended into full songs. 

One may ask why we choose to look at 15 to 30 options if all are reasonable.  Having a variety of options can lead to better quality songs while also enabling artists to incorporate their own musical preferences, without possessing the composition and text/melody juxtaposition skills traditionally required to engage in this art form.

When making our selections, we look for melodies that are independently interesting, and have intriguing relationships with the underlying text. We search for lines that match the emotional meaning of the text (happy or sad), as well as interesting word emphasis. For example, if the word ``sunshine'' appears in the text, we may select a melody that rises with this word (see Figure~\ref{fig:Rainbows}). ALYSIA often suggests melodic variations for which we find interesting explanations. For example, in the original song \emph{Why Do I Still Miss You} (see Figure~\ref{fig:WhyDoIScore}), ALYSIA suggested a melody where the phrase ``went wrong'' is on the lower end of scale, giving these words an appropriately dark interpretation. 

The next step involves combining the melodic lines to form the complete song. This takes some trial and error, and several interesting variations can typically be attained. The most efficient approach we had found was to construct the song around a few melodic lines. Say, if there are particularly interesting melodies for lines 1 and 3, then lines 2 and 4 are chosen to fit with these. Having multiple options allows the artist to create several variations for verses and chorus repetitions, as often found in songs. 

\noindent%
\begin{minipage}{\linewidth}%
\makebox[\linewidth]{
%   \advance\leftskip-0.2cm
    \includegraphics[width=11cm]{firstline.jpg}}
    \vspace{-15mm}
    \captionof{figure}{The first 12 melodies generated by ALYSIA to accompany the first line of the song \emph{Everywhere} (the full song appears in the following section). All melodies are sensible and could be used towards the creation of a complete melody line. Having many reasonable options allows the user to incorporate their musical taste into songwriting with ALYSIA.}
    \label{fig:variations}
\end{minipage}
%\vspace{2mm}




\newpage
\section{Songs made with ALYSIA}

Three songs were created using ALYSIA. The first two rely on lyrics written by the authors, and the last borrows its lyrics from a 1917 Vaudeville song. The songs were consequently recorded and produced. All recordings are posted on \href{http://bit.ly/2eQHado}{http://bit.ly/2eQHado}.

The primary technical difference between these songs was the explore/exploit parameter. \emph{Why Do I Still Miss You} was made using an explore vs exploit parameter that leads to more exploration, while \emph{Everywhere} and \emph{I'm Always Chasing Rainbows} were made with a higher exploit setting. 


%The creation of an original song began with the creation of a corpus. ALYSIA was trained on.... (SEE ABOVE). 

%Discuss the parameters, how the explore was set to high. 
\subsection{\emph{Why Do I Still Miss You}}

\emph{Why Do I Still Miss You}, the first song ever written with ALYSIA, was made with the goal of testing the viability of using computer-assisted composition for production-quality songs. The creation of the song started with the writing of original lyrics, written by the authors.  The goal was to come up with lyrics that stylistically match the style of the corpus, that is, lyrics in today's pop genre (See Figure~\ref{fig:WhyDoIScore}). 

%Aiming for the modern remix style, so that they could be explored through varied accompaniment styles as is commonly done when remixing a song.  

%The experiment went remarkably well, exceeding the authors' expectations.

%Here are the lyrics:

%\vspace{3mm}

%\noindent \emph{Now that you're gone,}\\
%\emph{I just realized, I'm all alone.}\\
%\emph{Forgive me if I, throw away the phone.}\\
%\emph{Stop wondering where we went wrong.}\\

%Chorus:\\
%\emph{Tell me, after all you've done}\\
%\emph{Why do I still miss you?}\\
%\emph{Why, do I still miss you.}
%\vspace{3mm}


Next, each lyrical line was given to ALYSIA, for which she produced between fifteen and thirty variations. In most cases, we were satisfied with one of the options ALYSIA proposed within the first fifteen tries.  We then combined the melodic lines. See Figure~\ref{fig:WhyDoIScore} for the resulting score. Subsequently, the song was recorded and produced. One of the authors, who is a professionally trained singer, recorded and produced the song. A recording of this song can be heard at \href{http://bit.ly/2eQHado}{http://bit.ly/2eQHado}. 

\vspace{4mm}
\noindent%
\begin{minipage}{\linewidth}%
\makebox[\linewidth]{
    %\advance\leftskip-1cm
    \includegraphics[width=10cm]{WhyDoIStillMissYou-short.jpg}}
    \vspace{-3mm}
    \captionof{figure}{An original song co-written with ALYSIA.}
    \label{fig:WhyDoIScore}
\end{minipage}
\vspace{2mm}



%The recording of this song starts with a short melodic introduction based on ALYSIA's accompaniments,  followed by the combined melody and lyrics. It is then followed by fragments that emphasize either the lyrics or the melody, in order to better expose the quality of the underlying melody, as well as how well it works with the human voice without instrumental accompaniment. 



%XXX SUPPLEMENTARY MATERIAL FOR MUSIC?

This experiment, intentionally spanning the entire songwriting process from the lyrics to the final production, demonstrates the viability of our approach for making production-worthy works. It also shows that ALYSIA makes songwriting accessible to those with no songwriting ability, as the song was written by one of the authors, who does not possess this skill unassisted by our computer system. Informal assessment by the authors and many others consistently point to the surprising quality of this song, gauging it as having quality on par with that of fully human-made songs. Future user studies will help to formally assess this claim. 

For this song, we used an explore/exploit parameter of 1, meaning that the distribution of notes output by the model was sampled only once. With regard to improving the application of our system, the making of \emph{Why Do I Still Miss You} suggested that we should increase the exploit parameter. This was particularly important for the rhythm suggestion, which we found to be too varied. As such, \emph{Why Do I Still Miss You} fully utilized the melody model, while making some manual adjustment to the rhythm. We stress that even with this sub-optimal setting for the exploit-explore setting, ALYSIA made songwriting accessible to someone who previously could not engage in this artform.  

The two subsequent songs discussed below exploit our model more heavily, using sampling four times for the rhythm model and two times for the melody model, selecting the majority, which corresponds to an explore/exploit parameter of 4 and 2, respectively. Heavier reliance on the model allowed us to use the rhythm suggestions without alterations and also raised the proportion of high quality scale degree sequences. 


\subsection{\emph{Everywhere}}\label{sec:Everywhere}

\emph{Everywhere} is the first original song written with ALYSIA using a high setting of the exploit parameter. This made it even easier to discover new melodies. In fact, so many of ALYSIA's melodies were interesting that combining melodies became more time consuming (of course, one could simply consider a smaller set of melodies per line to overcome this challenge). Nevertheless, the writing of \emph{Everywhere} from start to finish took only about two hours. Figure~\ref{fig:everywhere} shows the score co-created with ALYSIA. A recording appears on \href{http://bit.ly/2eQHado}{http://bit.ly/2eQHado}

%- we stress once again that it was written by Margareta who up until the development of ALYSIA was unable to write original songs. 

%The lyrics were written in a similar style and on a similar subject to the previous song. 

 %Please check \href{http://www.cs.fsu.edu/~ackerman/ALYSIA.html}{http://www.cs.fsu.edu/$\sim$ackerman/ALYSIA.html}
 %for a recording of \emph{Everywhere}, scheduled to complete by March 14, 2016. 

%\vspace{2mm}
%\noindent\emph{Everywhere I look I find you looking back}\\
%\emph{On the side walk, baby}\\
%\emph{And the leaves reflecting in the lake}\\
%\emph{In an old man's smile}\\
%\emph{In a young boy’s shadow}\\
%\emph{Why does everything remind me of you?}\\

%\noindent Chorus:\\
%\emph{I miss your voice}\\
%\emph{Oh, how I wish I had a choice}\\
%\emph{To turn back time}\\
%\emph{Back, to when you were mine}\\

 
 \subsection{\emph{I'm Always Chasing Rainbows}}

%ALYSIA's fully data-driven model and the absence of directly encoded musical rules make it an ideal platform for genre translation. 

The final song made with ALYSIA so far relies on lyrics from the song \emph{I'm Always Chasing Rainbows}. The original music is credited to Harry Carroll, although the melody was in fact adapted from Chopin's Fantaisie-Impromptu. Lyrics were written by  Joseph McCarthy. The song's appearance in Broadway musicals and motion pictures contributed to its popularity. As it was published in 1917, \emph{I'm Always Chasing Rainbows} is in public domain. 

Due to its Vaudeville's style and classical roots, we thought it would be interesting to recreate \emph{I'm Always Chasing Rainbows} as a pop song. ALYSIA's version is given in Figure~\ref{fig:Rainbows}. A production of ALYSIA's pop version of this Vaudeville song has been posted on \href{http://bit.ly/2eQHado}{http://bit.ly/2eQHado}, where you can also find the original score. 

\noindent%
\begin{minipage}{\linewidth}%
\makebox[\linewidth]{
%   \advance\leftskip-1.5cm
    \includegraphics[width=9cm]{Everywhere.jpg}}
    \vspace{-15mm}
    \captionof{figure}{An original song co-written with ALYSIA.}
    \label{fig:everywhere}
\end{minipage}
%\vspace{2mm}


\section{Discussion: Co-Creative and Autonomous Songwriting}

%Computational Creativity (CC) is concerned with both autonomous and human-assisted creativity, and although this is a continuum, weak and strong CC have some distinct objectives. 

Songwriting is the art of combining melodies and lyrics; It is not enough to have beautiful melodies and poetic lyrics, the music and words must fit together into a coherent whole. This makes Algorithmic Songwriting a distinct sub-field of Algorithmic Composition. ALYSIA is the first machine-learning system that learns the relationship between melodies and lyrics, and uses the resulting model to create new songs in the style of the corpus. 

The unique challenges of songwriting were observed during the creation of the first musical to rely on computational creativity systems, Beyond the Fence, which played in London in the early months of 2016~\cite{jordanous2016has}. During the panel on  Beyond the Fence held at the Seventh International Conference of Computational Creativity, it was noted that the juxtaposition of music and text posed a substantial challenge in the making of this musical. This further illustrates the need for systems that focus on uncovering the complex relationships between music and lyrics. 


\vspace{-1mm}
\noindent%
\begin{minipage}{\linewidth}%
\makebox[\linewidth]{
        %\advance\leftskip-1cm
    \includegraphics[width=9cm]{ChasingRainbows-ALYSIA.jpg}}
    %\vspace{-8mm}
    \captionof{figure}{Melody and rhythm by ALYSIA, lyrics by Joseph McCarthy from the popular Vaudeville song \emph{I'm Always Chasing Rainbows}.}
\label{fig:Rainbows}
\end{minipage}
\vspace{2mm}
   
%\begin{figure}
%        \advance\leftskip-1cm
%    \includegraphics[width=10cm]{Rainbows_original.jpg}
%    \caption{The original score for \emph{I'm Always Chasing Rainbows}, lyrics by Joseph McCarthy and melody by Harry Correll.}
%\label{fig:Original_rainbows}
%\end{figure}
   


Algorithmic songwriting offers intriguing challenges as both an autonomous and a co-creative system. An autonomous songwriting system producing works on par with those of expert human songwriters would mark a significant achievement. Yet, we can go beyond the score. What if, in addition to writing the song, an automated system could also perform and record its own compositions? A truly independent system would not only create a score, but incorporate the full spectrum of expertise required for the creation of a complete song, including the vocal performance, expressive rendition, and automated music production. 



%Few possess the skills required for the multifaceted art of song-making. This is where the role of co-creative CC comes in. Our systems can help musicians at varying degrees of expertise explore their creative space and find new musical works that they could not have created independently. 

As we aspire to create autonomous songwriters, artists and hobbyists alike are thirsty for our help. 
Even if we had access to a fully autonomous songwriter, it would not replace the need for a corresponding co-creative system. Whereas an autonomous songwriter could be used when complete works are desired, a co-creative variation would satisfy the human need for music making - much like the difference between the joys of eating and cooking. A co-creative algorithmic songwriter would expand our creative repertoire, making songwriting accessible to those who cannot otherwise enjoy this art-form, and be of great use to semi-professional and amateur musicians (particularly singers) who do have the luxury of hiring a songwriter. 

 In the development of a co-creative songwriting system, it is desired that the users retain creative control, allowing them to claim ownership of the resulting works, or at least experience the process as an expression of creativity. The goal is to relieve the burden of having to master all of the diverse skills needed for the creation of a song, giving users the freedom to focus on aspects of the creative process in which they either specialize or find most enjoyable. 
 
 %ALYSIA will relieve the burden of having to master all of the diverse skills needed for the creation of a song, giving users the freedom to focus on aspects of the creative process in which they either specialize or find most enjoyable. 

%Currently, ALYSIA takes care of one of the most challenging parts of songwriting: coming up with interesting melodies to accompany lyrics. Our system makes it possible to write songs for those who otherwise cannot engage in this creative activity, while allowing for sufficient user input to express their musical preferences.  

From a technical viewpoint, the goals of autonomous and co-creative algorithmic songwriters converge - the primary difference being that a co-creative system allows for more human interaction. ALYSIA is a long-term project in which we will simultaneously explore both of these paths as the system's technical foundation continues to expands. 


 
 %will help to further flesh out our songwriting system. 

%It has been argued that any melody can lead to pleasing music if accompanied with well-chosen chords. However, we emphasize that w

Short-term goals for ALYSIA include the automation of chord progressions that would underlie the melody. A couple of user studies are being planned where people at different levels of musical expertise will evaluate the songs made with ALYSIA. To further our machine learning methodology, we are putting together a corpora of songs in Music XML format. For several different genres, a large corpus will be used to train a model using a neural network, which requires large data sets to avoid over-fitting. This will also let us explore differences in melodies resulting from different machine learning models.  We are also experimenting with our system's potential for songwriting in different genres and languages, ranging from English pop songs discussed here to the creation of new arias in the style of classical Italian composer Giacomo Puccini. 

\vspace{-2mm}
\bibliographystyle{plain}
\bibliography{references}


%Other short-term goals include a couple of user studies. The first will evaluate the quality of the songs against those made by humans. 

%Music is subjective. It is therefore helpful to evaluate the songs produced with ALYSIA against those made by humans at different levels of expertise in a blind study where the judges have no information on who created the music. 

%Another study will explore the utility of ALYSIA to users at different levels of musical expertise. We  suspect that ALYSIA will prove beneficial at all levels, helping musical novices make reasonable songs and allowing professionals to more easily explore the space of melodic possibilities.  From a user-interface viewpoint, we conjecture that the complexity of the interface should increase with the degree of musical expertise, allowing those who are more musically advanced to incorporate their domain knowledge, while making the interface minimal for those with no musical experience.

%Models trained using different musical genres will allow for a deeper exploration of genre translation, as well as the creation of original songs across different styles. We are particularly excited by the upcoming application of ALYSIA to the writing of R\&B music by a professional songwriter.  


%\section{Why melodies?}

%One may ask whether the problem we are trying to address can be solved with simple random generation. For example, it can be argued that \emph{any} melody can sound good if  placed within a well-chosen harmonic context\cite{papadopoulos1999ai}. 

%How can we expect to
%evaluate the musical output if we do not have a harmonic
%context for it? Most melodies will sound acceptable in
%some context or other.

%We emphasize that our goal here is merely not create pleasing melodies (although that, too, is  worthy goal), but to create vocal melodies that fit with the underlying lyrics and fall within a particular musical genre. 

%To this end, the problem becomes challenging for humans and computers alike. To illustrate just one of the many challenges involved in this task, consider the sentence ``Rainbows paint the sky tonight.'' Say we choose a melody where the syllable ``bows'' is emphasized in the rhythm by assigning it a notable longer note (say a half note) than the other notes (for example, eighth notes). Even if the melody is highly pleasing on its own, singing it with the given lyrics will sound artificial, because it would violate the word emphasis in ``rainbows,'' where ``rain'' is the stressed syllable. As such, the rhythm needs to respect the underlying language. 

%In addition, there are special considerations that apply to limitations of the human voice. Most singers (even professionals) have a limited comfortable vocal range. This is even further constrained when we focus on specific genres (for example, opera arias make greater use of higher notes than pop songs). When we wish to discover melodies that fit specific genres, many other factors came into play. For another example, the occurrence of consecutive repeated notes is notable more common in pop than in opera. 

%Whether conscious or unconscious, composers and songwriters incorporate their knowledge of the linguistics and specific musical genres when writing songs. When applying machine learning, we allow ALYSIA to learn these rules from the data. 

%Interestingly, this can be achieved even when  we, its creators, don't have complete knowledge of how to combine and utilize the features we give to the system. 

% There are two prominent approaches to composition: One involves starting with the melody, and the other begins with a chord progression and then identifies a suitable melody. Both are legitimate methods, and the choice ultimately boils down to personal preference. There are several reasons we chose to focus on melody generation. When it comes to songs that include lyrics, the melody plays a particularly important role. With rare exceptions, singers can only sing a single note at a time. In addition, the words sang by the singer tend to stand out from the remaining music, making the melody highly prominent. Further, when we incorporate the requirement that the song should fall within a specific genre, as we do here, the melody becomes even more essential. For example, consecutive note repetitions occur with high frequency in pop songs, than, say, in opera arias, and range varies notably between these genres. ALYSIA currently produces melodies, and we plan to incorporate automated chord generation in future work.


%\bibliographystyle{aaai} 


\end{document}

main.bbl
\begin{thebibliography}{1}

\bibitem{fernandez2013ai}
Jose~D Fern{\'a}ndez and Francisco Vico.
\newblock Ai methods in algorithmic composition: A comprehensive survey.
\newblock {\em Journal of Artificial Intelligence Research}, pages 513--582,
  2013.

\bibitem{gauntlett2013making}
David Gauntlett.
\newblock {\em Making is connecting}.
\newblock John Wiley \& Sons, 2013.

\bibitem{jordanous2016has}
Anna Jordanous.
\newblock Has computational creativity successfully made it ‘beyond the
  fence’in musical theatre?
\newblock In {\em Proceedings of the 7th International Conference on
  Computational Creativity}, 2016.

\bibitem{monteith2012automatic}
Kristine Monteith, Tony Martinez, and Dan Ventura.
\newblock Automatic generation of melodic accompaniments for lyrics.
\newblock In {\em Proceedings of the International Conference on Computational
  Creativity}, pages 87--94, 2012.

\bibitem{nichols2009lyric}
Eric Nichols.
\newblock {\em Lyric-based rhythm suggestion}.
\newblock Ann Arbor, MI: Michigan Publishing, University of Michigan Library,
  2009.

\bibitem{oliveira2015tra}
Hugo~Gon{\c{c}}alo Oliveira.
\newblock Tra-la-lyrics 2.0: Automatic generation of song lyrics on a semantic
  domain.
\newblock {\em Journal of Artificial General Intelligence}, 6(1):87--110, 2015.

\bibitem{scirea2015smug}
Marco Scirea, Gabriella~AB Barros, Noor Shaker, and Julian Togelius.
\newblock Smug: Scientific music generator.
\newblock In {\em Proceedings of the Sixth International Conference on
  Computational Creativity June}, page 204, 2015.

\bibitem{toivanen2013automatical}
Jukka~M Toivanen, Hannu Toivonen, and Alessandro Valitutti.
\newblock Automatical composition of lyrical songs.
\newblock In {\em The Fourth International Conference on Computational
  Creativity}, 2013.

\end{thebibliography}
