\section{Introduction}


  The recent progress in deep learning for sequence generation has
  led to the emergence of audio synthesis systems that directly
  generate the waveform, reaching state-of-the-art perceptual quality in
  speech synthesis, and promising results for music generation. This
  represents a shift of paradigm with respect to
  approaches that generate sequences of parameters to
  vocoders in text-to-speech systems \cite{char2wav,voiceloop,ping2018deep}, or MIDI partition in music
  generation \cite{deepbach,expertmusic,morpheus}.
  A commonality between the state-of-the-art neural audio
  synthesis models is the use of discretized sample values, so that
  an audio sample is predicted by a categorical
  distribution trained with a classification loss
  \cite{wavenet,samplernn,parallel_wavenet,kalchbrenner2018efficient}. Another
  significant commonality is the use of autoregressive models that generate
  samples one-by-one, which leads to prohibitive training and inference
  times \cite{wavenet,samplernn}, or requires specialized
  implementations and low-level code optimizations to run in real time
  \cite{kalchbrenner2018efficient}. An exception is parallel WaveNet
  \cite{parallel_wavenet} which generates a sequence with a fully
  convolutional network for faster inference. However, the parallel
  approach is trained to reproduce the output of a standard WaveNet,
  which means that faster inference comes at the cost of increased
  training time.

  In this paper, we study an alternative to both the modeling of audio
  samples as a categorical distribution and the autoregressive
  approach. We propose to generate the waveform for entire
  audio frames of $1024$ samples at a time with a large stride, and
  model audio samples as continuous values. We develop and evaluate
  this method on the challenging task of generating musical notes
  based on the desired instrument, pitch, and velocity, using the large-scale
  NSynth dataset \cite{nsynth}. We obtain a lightweight synthesizer of
  musical notes composed of a 3-layer RNN with LSTM cells \cite{lstm}
  that produces embeddings of audio frames given
  the desired instrument, pitch, velocity\footnote{Quoting \cite{nsynth}:
  "MIDI velocity is similar to volume control and they have a direct relationship.
  For physical intuition, higher velocity corresponds to pressing a piano key harder."}
  and time index. These
  embeddings are decoded by a single four-layer convolutional network
  to generate notes from nearly $1000$ instruments, $65$ pitches per
  instrument on average and $5$ velocities.

  The successful end-to-end training of the
  synthesizer relies on two ingredients:
  \begin{itemize}[wide=0pt,leftmargin=\dimexpr\labelwidth + 2\labelsep\relax]
  \item A new loss function which we call the
    \emph{spectral loss}, which computes the $1$-norm between the log
    power spectrograms of the waveform generated by the model and the
    target waveform, where the power spectrograms are obtained by the
    short-time Fourier transform (STFT).

    Log power spectrograms are interesting both because they are related
    to human perception \cite{goldstein1967auditory}, but more
    importantly because the entire loss is invariant to the original
    phase of the signal, which can be arbitrary without audible
    differences.
  \item Initialization with a pre-trained autoencoder:
    a purely convolutional autoencoder architecture on raw waveforms
    is first trained with the spectral loss.
    The LSTM is then initialized to reproduce the embeddings given by the encoder,
    using mean squared error. After initialization, the LSTM and the decoder
    are fine-tuned together, backpropagating through the spectral loss.
  \end{itemize}

  We evaluate our synthesizer on a new task of pitch completion:
  generating notes for pitches not seen during training. We perform
  perceptual experiments with human evaluators to aggregate a Mean
  Opinion Score (MOS) that characterizes the naturalness and appealing
  of the generated sounds. We also perform ABX tests to measure the
  relative similarity of the synthesizer's ability to effectively
  produce a new pitch for a given instrument, see Section~\ref{sec:abx}.
  We use a state-of-the-art
  autoencoder of musical notes based on WaveNet \cite{nsynth} as a
  baseline neural audio synthesis system.
  Our synthesizer achieves higher perceptual quality than Wavenet-based autoencoder
  in terms of MOS and similarity to the
  ground-truth while being about $32$ times faster during
  training and $2,500$ times for generation.

  \section{Related Work}

  A large body of work in machine learning for audio synthesis focuses on
  generating parameters for vocoders in speech
  processing \cite{char2wav,voiceloop,ping2018deep} or musical instrument
  synthesizers in automatic music composition \cite{deepbach,expertmusic,morpheus}.
  Our goal is to learn the synthesizers for musical instruments,
  so we focus here on methods that generate sound without calling such synthesizers.

  A first type of approaches model power spectrograms given by the STFT
  \cite{nsynth,DBLP:journals/corr/abs-1804-00047,tacotron}, and generate the waveform
  through a post-processing that is not part of the training using
  a phase reconstruction algorithm such as the Griffin-Lim
  algorithm~\cite{griffin_lim}. The advantage is to focus on a distance between
  high-level representations that is more relevant perceptually than a regression
  on the waveform.
  However, using Griffin-Lim means that the training is not end to end.
  Indeed the predicted spectrograms may not come from a real signal.
  In that case, Griffin-Lim
  performs an orthogonal projection onto the set of
  valid spectrograms that is not accounted for during training.
  Notice that our approach with the spectral loss is different:
  our models directly predict waveforms rather than spectrograms
  and the spectral loss computes log power spectrograms of these predicted waveforms.

  The current state-of-the-art in neural audio synthesis is to generate directly
  the waveform \cite{wavenet,samplernn,ping2018deep}. Individual audio samples are modeled
  with a categorical distribution trained with a multiclass cross-entropy loss.
  Quantization of the 16 bit audio is performed (either linear \cite{samplernn} or
  with a $\mu$-law companding
  \cite{wavenet}) to map to a few hundred bins to improve scalability.
  The generation is still extremely costly; distillation \cite{hinton2015distilling}
  to a faster model has been proposed to reduce inference time at the expense of an even
  larger training time \cite{parallel_wavenet}. The recent proposal of \cite{kalchbrenner2018efficient}
  partly solves the issue with a small loss in accuracy, but it requires heavy low-level
  code optimization. In contrast, our approach trains and generate waveforms comparably
  fast with a
  PyTorch\footnote{\url{https://pytorch.org/}} implementation.
  Our approach is different since we model the waveform as a continuous signal and
  use the spectral loss between generated and target waveforms and model audio frames
  of $1024$ samples, rather than performing classification on individual samples.
  The spectral loss we introduce is also different from the power
  loss regularization of \cite{parallel_wavenet}, even though both are based on
  the STFT of the generated and target waveforms.
  In \cite{parallel_wavenet}, the primary loss is the classification of
  individual samples, and their power loss is used to equalize the average
  amplitude of frequencies over time. Thus the power loss cannot be used alone
  to learn to reconstruct the waveform.

  Works on neural audio synthesis conditioned on symbolic inputs were developed
  mostly for text-to-speech synthesis \cite{wavenet,samplernn,tacotron}.
  Experiments on generation of musical tracks based on desired properties
   were described in \cite{wavenet}, but no systematic evaluation has been published.
   The model of~\cite{nsynth}, which we use as baseline in our experiments on
   perceptual quality, is an autoencoder of musical notes based on
   WaveNet \cite{wavenet} that compresses the signal to generate high-level
   representations that transfer to music classification tasks,
   but contrarily to our synthesizer, it cannot be used to generate waveforms
   from desired properties of the instrument, pitch and velocity without some input signal.

  The minimization by gradient descent of an objective function based on the power
  spectrogram has already been applied to the transformation of a white noise waveform
  into a specific sound texture~\cite{caracalla2017gradient}. However, to the best
  of our knowledge, such objective functions have not been used in the context of
  neural audio synthesis.



  %
\section{The spectral loss for waveform synthesis}
\label{sec:losses}
Previous work in audio synthesis on the waveform focused on classification losses~\cite{samplernn,wavenet,nsynth}. However, their computational cost needs to be mitigated by quantization, which inherently limits the resolution of the predictions, and ultimately increasing the number of classes is likely necessary to achieve the optimal accuracy. Our approach directly predicts a single continuous value for each audio sample, and computes distances between waveforms in the domain of power spectra to be invariant to the original phase of the signal. As a baseline, we also consider computing distances between waveforms using plain mean square error (MSE).


\subsection{Mean square regression on the waveform}

The simplest way of measuring the distance between a reconstructed signal $\hat{x}$
and the reference $x$ is to compute the MSE on the waveform directly, that is taking the Euclidean norm between $x$ and $\hat{x}$,
\begin{equation}
\label{eq:loss_wav}
  \Lwav{x, \hat{x}} := \norm{x - \hat{x}}^2.
\end{equation}
The MSE is most likely not suited as a perceptual distance between waveforms because it is extremely sensitive to a small shift in the signal. Yet, we observed that it was sufficient to learn an autoencoder and use it as a baseline.

\subsection{Spectral loss}

As an alternative to the MSE on the waveform, we suggest taking the Short Term Fourier Transform (STFT)
of both $x$ and $\hat{x}$ and compare their absolute values in log scale. We first compute the log spectrogram
\begin{equation}
  \label{eq:log_spec}
  l(x) := \log\left(\epsilon + \abs{\STFT{x}}^2\right).
\end{equation}
The STFT decomposes the original signal $x$ in successive
frames of 1024 time steps with a stride of 256 so that a frame overlaps at 75\% with the next one.
The output for a single frame is given by 513 complex numbers, each representing a specific frequency range.
Taking the point-wise absolute values of those numbers represents how much energy is present in a specific
frequency range. We observed that our models generated higher quality sounds when trained
using a log scale of those coefficients. Previous work has come to the same conclusion~\cite{nsynth}.
We observed that many entries of the spectrograms are close to zero and that small errors on those parts can add up
to form noisy artifacts. In order to favor sparsity in the spectrogram, we use the $\norm{\cdot}_1$ norm instead of the MSE,
\begin{equation}
  \label{eq:loss_spec}
  \Lspec{x, \hat{x}} := \norm{l(x) - l(\hat{x})}_1.
\end{equation}
The value of $\epsilon$ controls the trade-off between accurately representing low energy and high energy coefficients in the spectrogram. We found
that $\epsilon=1$ gave the best subjective reconstruction quality.

The STFT is a (complex) convolution operator on the waveform and the squared absolute value of the Fourier coefficients makes the power spectrum differentiable with respect to the generated waveform. Since the generated waveform is itself a differentiable function of the parameters (up to the non-differentiability points of activation functions such as ReLU), the spectral loss \eqref{eq:loss_spec} can be minimized by standard backpropagation.
Even though we only consider this spectral loss in our experiments, alternatives to the STFT such as the Wavelet transform
also define differentiable loss for suitable wavelets.

\subsubsection{Non unicity of the waveform representation}
\label{sec:losses:nonunicity}

To illustrate the importance of the spectral loss instead of a waveform loss, let us now consider a problem that arises when generating notes in the test set.
Let us assume one of the instrument is a pure sinuoide. For a given pitch at a frequency $f$, the audio signal is
$x_i = \sin(2 \pi i \frac{f}{16000} + \phi)$.
Our perception of the signal is not affected by the choice of $\phi \in [0, 2 \pi[$, and the power spectrogram of $x$ is also unaltered.
When recording an acoustic instrument, the value of $\phi$ depends on any number of variables characterizing the physical system that
generated the sound and there is no guarantee that $\phi$ stays constant when playing the same note again. For a synthetic sound, $\phi$ also depends on implementation details of the software generating the sound.

For a sound that is not in the training set and as far as the model is concerned, $\phi$ is a random variable that can take any value in the range $[0, 2 \pi[$. As a result, $x_0$ is unpredictable in the range $[-1, 1]$, and the mean square error between the generated signal and the ground truth is uninformative. Even on the training dataset, the model has to use extra resources to remember the value of $\phi$ for each pitch. We believe that this phenomenon is the reason why   training the synthesizer using the MSE on the waveform leads to worse reconstruction performance, even though this loss is sufficient in the context of auto-encoding (see Section~\ref{sec:pitch_completion}).
The spectral loss solves this issue since the model is free to choose a single canonical value for $\phi$.

However, one should note that the spectral loss is permissive, in the sense that it does not penalize phase inconsitencies
of the complex phase across the different frames of the STFT, which lead to potential artifacts. In practice,
we obtain state of the art results (see Section \ref{sec:experiments}) and we conjecture that thanks to the frame overlap
in the STFT, the solution that minimizes the spectral loss will often be phase consistent, which is why Griffin-Lim works resonably well
despite sharing the same limitation.

  %
\section{Model}

\begin{figure}
\centering
\begin{tikzpicture}[
      every node/.style={font=\tiny},
      subworker/.style={rounded corners=1pt,rectangle,draw=black!90,fill=black!10,minimum width=1cm, minimum height=1cm},
      rnn/.style={rounded corners=1pt,rectangle,draw=black!90,fill=blue!5,minimum width=0.6cm, minimum height=0.6cm},
      conv/.style={rounded corners=1pt,rectangle,draw=black!90,fill=gray!5,minimum width=1cm},
      stft/.style={rounded corners=1pt,rectangle,draw=black!90,fill=black!5!blue,minimum width=1cm},
      emb/.style={},
      myarrow/.style={->,>=stealth}]
    ]
  \begin{scope}
        
\node(h0) at (-1.3, 0) {$h_0 := 0$};

\node[yshift=-0.85cm](input1) at (0,0) {$(u_V,v_I,w_P,z_1)$};
\node(rnn1) at (0,0) [rnn] {};
\draw[myarrow] (input1) -- (rnn1);
\draw[myarrow] (h0) -- (rnn1);

\node[yshift=0.85cm](emb1) at (0,0) [emb] {$s_1(VIP)$};
\draw[myarrow] (rnn1) -- (emb1);

\node[yshift=-0.85cm](input2) at (2,0) {$(u_V,v_I,w_P,z_2)$};
\node(rnn2) at (2,0) [rnn] {};
\draw[myarrow] (input2) -- (rnn2);
\draw[->] [anchor=east] (rnn1) -- (rnn2) node [anchor=south,midway] {$h_1$};

\node[yshift=0.85cm](emb2) at (2,0) [emb] {$s_2(VIP)$};
\draw[myarrow] (rnn2) -- (emb2);


\node[yshift=-0.85cm](input3) at (4,0) {$(u_V,v_I,w_P,z_3)$};
\node(rnn3) at (4,0) [rnn] {};
\draw[myarrow] (input3) -- (rnn3);
\draw[->] [anchor=east] (rnn2) -- (rnn3) node [anchor=south,midway] {$h_2$};

\node[yshift=0.85cm](emb3) at (rnn3) [emb] {$s_3(VIP)$};
\draw[myarrow] (rnn3) -- (emb3);

\node(rnndots) at (6,0) [rnn] {$\cdots$};
\draw[myarrow] (rnn3) -- (rnndots);

  \node[yshift=-0.85cm](input4) at (8,0) {$(u_V,v_I,w_P,z_{265})$};
  \node(rnn4) at (8,0) [rnn] {};
  \draw[myarrow] (input4) -- (rnn4);
  \draw[->] [anchor=east] (rnndots) -- (rnn4) node [anchor=south,midway] {$h_{259}$};

  \node[yshift=0.85cm](emb4) at (rnn4) [emb] {$s_{265}(VIP)$};
  \draw[myarrow] (rnn4) -- (emb4);


        \draw (emb1.north) ++(0,0.1cm) -- ++(0.5cm, 0.4cm) -- ($ (emb4.north)+(-0.5cm, 0.5cm)$) -- ($ (emb4.north) + (0, 0.1cm)$) -- ($ (emb1.north) + (0, 0.1cm)$);
\begin{scope}[yshift=-0.3cm]
    \node(conv1) at (4,2.2) [conv] {Convolution $K=9$, $S=1$, $C=4096$, ReLU};
    \node(conv1x1) at (4,2.8) [conv] {Convolution $K=1$, $S=1$, $C=4096$, ReLU};
    \draw[myarrow] (conv1) -- (conv1x1);
      \node(conv1x2) at (4,3.4) [conv] {Convolution $K=1$, $S=1$, $C=4096$, ReLU};
    \draw[myarrow] (conv1x1) -- (conv1x2);
            \node(convt) at (4,4.0) [conv] {Conv transpose $K=1024$, $S=256$, $C=1$};
            \draw[myarrow] (conv1x2) -- (convt);
    \draw (convt.north east) ++(0,0.1cm) -- ++(0.5cm, 0.4cm) --
      ($ (convt.north west)+(-0.5cm, 0.5cm)$) --  ($ (convt.north west)+(-0.5cm, 0.5cm)$) --
      ($ (convt.north west)+(0cm, 0.1cm)$) -- ($ (convt.north east)+(0cm, 0.1cm)$);;

\begin{scope}[yshift=4.7cm, xshift=-0.2cm,local bounding box=scope1]
  \begin{axis}[domain=0:20,
  axis y line=none,
  axis x line=none,
  samples=200,
  height=2.5cm,
  width=10cm]
    \addplot[mark=none,black!20!blue] {cos(deg(2 * x))};
    \end{axis}
\end{scope}
\end{scope}


        \node(out) at ($(scope1.north) + (-2cm, 0.5cm)$) [conv] {Output waveform};

        \node(stft) at ($(scope1.north) + (2cm, 0.5cm)$) [conv] {STFT + log-power};
        \draw[myarrow] (scope1.north) -- ++(0, 0.5) -- (stft.west);
              \draw[myarrow] (scope1.north) -- ++(0, 0.5) -- (out.east);
        \node(loss) at ($(stft) + (0cm, 0.8cm)$) [conv] {Spectral loss};
        \draw[myarrow] (stft) -- (loss);

  \end{scope}

    \end{tikzpicture}

  \caption{\label{figure:sing} Summary of the entire architecture of SING. $u_V, v_I, w_P, z_*$ represent the look-up tables respectively for the velocity, instrument, pitch and time. $h_*$ represent the hidden state of the LSTM and $s_*$ its output. For convolutional layers, $K$ represents the kernel size, $S$ the stride and $C$ the number of channels.}
\end{figure}


In this section we introduce the SING architecture. It is composed of two parts: a LSTM based sequence generator whose output is plugged to a
decoder that transforms it into a waveform.
The model is trained to recover a waveform $x$ sampled at 16,000 Hz from the training set based on the one-hot encoded instrument $I$, pitch $P$ and velocity $V$.
The whole architecture is summarized in Figure~\ref{figure:sing}.

\subsection{LSTM sequence generator}
The sequence generator is composed of a 3-layer recurrent neural network with LSTM cells and 1024 hidden units each.
Given an example with velocity $V$, instrument $I$ and pitch $P$, we obtain
3 embeddings $(u_V, v_I, w_P) \in \reel^{2}\times\reel^{16}\times\reel^{8}$ from look-up tables that are trained along with the model.
Furthermore, the model is provided at each time step with an extra embedding $z_T \in\reel^{4}$ where $T$ is the current
time step~\cite{memnet, convmt}, also obtained from a look-up table that is trained jointly. The input of the LSTM is the concatenation
of those four vectors $(u_V, v_I, w_P, z_T)$. Although we first experimented with an autoregressive model
where the previous output was concatenated with those embeddings,
we achieved much better performance and faster training by feeding the LSTM with only on the 4 vectors $(u_V, v_I, w_P, z_T)$ at each time step. Given those inputs, the recurrent network generates a sequence
$\forall 1\leq T \leq N, s(V, I, P)_T \in \reel^D$ with a linear layer on top of the last hidden state. In our experiments, we have $D = 128$ and $N=265$.

\subsection{Convolutional decoder}

The sequence $s(V,I,P)$ is decoded into a waveform by a convolutional network. The first layer is a convolution with a kernel size of 9 and a stride of 1 over the sequence $s$ with 4096 channels followed by a ReLU. The second and third layers are both convolutions with a kernel size of 1 (a.k.a. 1x1 convolution \cite{nsynth})
also followed by a ReLU. The number of channels is kept at 4096.
Finally the last layer is a transposed convolution with a stride of 256 and a kernel size of 1024 that directly outputs the final waveform corresponding to an audio frame of size $1024$.
In order to reduce artifacts generated by the high stride value, we smooth the deconvolution
filters by multiplying them with a squared Hann window. As the stride is one fourth
of the kernel size, the squared Hann window has the property that the sum of its values for
a given output position is always equal to 1 \cite{griffin_lim}.
Thus the final deconvolution can also be seen as an overlap-add method.
We pad the examples
 so that the final generated audio signal has the right length.
Given our parameters, we need $s(V,I,P)$ to be of length $N=265$ to recover a 4 seconds signal $d(s(V,I,P)) \in \reel^{64,000}$.


\subsection{Training details}

All the models are trained on 4 P100 GPUs using Adam \cite{adam} with a learning rate of 0.0003 and a batch size of 256.

\paragraph{Initialization with an autoencoder.}

We introduce an encoder turning a waveform $x$ into a sequence $e(x) \in \reel^{N\times D}$.
This encoder is almost the mirror of the decoder. It starts with a convolution layer
with a kernel size of 1024, a stride of 256 and 4096 channels followed by a ReLU. Similarly
to the decoder, we smooth its filters using a squared Hann window. Next are two 1x1 convolutions
with 4096 channels and ReLU as an activation function. A final 1x1 convolution with no non linearity turns those 4096 channels into the desired sequence with $D$ channels.
We first train the encoder and decoder together as an auto-encoder on a reconstruction task. We train the auto-encoder for 50 epochs which takes about 12 hours on 4 GPUs.

\paragraph{LSTM training.}

Once the auto-encoder has converged, we use the encoder to generate a target sequence for the LSTM. We use the MSE between the output $s(V,I,P)$ of the LSTM and the output $e(x)$ of the encoder, only optimizing the LSTM while keeping the encoder constant.
The LSTM is trained for 50 epochs using truncated backpropagation through time~\cite{williams1995gradient} using a sequence length of~32. This takes about 10 hours on 4 GPUs.

\paragraph{End-to-end fine tuning.}

We then plug the decoder on top of the LSTM and fine tune them together in an end-to-end
fashion, directly optimizing for the loss on the waveform, either using the MSE on the waveform or computing the MSE on the log-amplitude spectrograms and back propagating through the STFT. At that point we stop using truncated back propagation through time
and directly compute the gradient on the entire sequence. We do so for 20 epochs which takes about 8 hours on 4 GPUs.
From start to finish, SING takes about 30 hours on 4 GPUs to train.

Although we could have initialized our LSTM and decoder randomly and trained end-to-end, we did not achieve convergence until we implemented our initialization strategy.




  %

\section{Experiments}
\label{sec:experiments}

The source code for SING and a pretrained model are available on our
github\footnote{\url{https://github.com/facebookresearch/SING}}. Audio samples
are available on the article webpage\footnote{\url{https://research.fb.com/publications/sing-symbol-to-instrument-neural-generator}}.


\subsection{NSynth dataset}
\label{sec:nsynth}

The train set from the NSynth dataset~\cite{nsynth} is composed of 289,205 audio recordings of
instruments, some synthetic and some acoustic. Each recording is 4 second long at 16,000 Hz and is represented by a vector $x_{V, I, P}\in [-1, 1]^{64,000}$ indexed by
$V \in \{0, 4\}$ representing the velocity of the note,
$I \in \{0, \ldots, 1005\}$ representing the instrument,
 $P \in \{0, \ldots, 120\}$ representing the pitch.
The range of pitches available can vary depending on the instrument but for any
combination of $V, I, P$, there is at most a single recording.

We did not make use of the validation or test set from the original NSynth dataset because the instruments had no overlap with the training set. Because we use a look-up table for the instrument embedding, we cannot generate audio for unseen instruments.
Instead, we selected for each instrument 10\% of the pitches randomly that we moved to a separate test set. Because the pitches are different for each instrument, our model trains on all pitches but not on all combinations of a pitch and an instrument. We can then evaluate the ability
of our model to generalize to unseen combinations of instrument and pitch. In the rest of the paper, we refer to this new split of the original train set as the train and test set.


\subsection{Generalization through pitch completion}
\label{sec:pitch_completion}

We report our results in Table~\ref{table:completion_metrics}. We provided both the performance of the complete model as well as that of the autoencoder used for the
initial training of SING. This autoencoder serves as a reference for the maximum
quality the model can achieve if the LSTM were to reconstruct perfectly
the sequence $e(x)$.

Although using the MSE on the waveform works well as far as the autoencoder is concerned, this loss is hard to optimize for the LSTM.
Indeed, the autoencoder has access to the signal it must reconstruct, so that
it can easily choose which \emph{representation} of the signal to output as explained in Section~\ref{sec:losses:nonunicity}. SING must be able to recover that information solely from the embeddings given to it as input. It manages to learn some of it
but there is an important drop in quality. Besides, when switching to the test set one can see that the MSE on the waveform increases significantly. As the model has never seen
those examples, it has no way of picking the right representation.
When using a spectral loss, SING is free to choose a \emph{canonical} representation for the signal it has to reconstruct and it does not have to remember the one that was in the training set. We observe that although we have a drop in quality between the train and test set, our model is still able to generalize to unseen combinations of pitch and instrument.

Finally, we tried training a model without the time embedding $z_T$. Theoretically, the LSTM
could do without it by learning to count the number of time steps since the beginning of the sequence. However we do observe a significant drop in performance when removing this embedding, thus motivating
our choice.

On Figure~\ref{figure:spectros}, we represented the rainbowgrams for a particular example from the test set
as well as its reconstruction by the Wavenet-autoencoder, SING trained with the spectral and waveform loss and SING without time embedding. Rainbowgrams are defined in \cite{nsynth} as ``a CQT spectrogram with intensity
of lines proportional to the log magnitude of the power spectrum and color
given by the derivative of the phase''. A different derivative of the phase will lead to audible deformations
of the target signal. Such modification are not penalized by our spectral loss as explained in Section~\ref{sec:losses:nonunicity}. Nevertheless, we observe a mostly correct reconstruction of the derivative of the phase using SING.
More examples from the test set, including the rainbowgrams and audio files
are available on the article webpage\footnote{
\url{https://research.fb.com/publications/sing-symbol-to-instrument-neural-generator}}.


\begin{table*}
\begin{center}
  \setlength\extrarowheight{2pt}
\begin{tabular}{|l|l||c|c|c|c|}
  \cline{3-6}
  \multicolumn{2}{c}{}& \multicolumn{2}{|c|}{\textbf{Spectral loss}} & \multicolumn{2}{c|}{\textbf{Wav MSE}}\\
  \hline
  \textbf{Model}& \textbf{training loss} & {\emph{train}} & {\emph{test}} &  {\emph{train}}& {\emph{test}} \\
  \hline \hline
\makecell[l]{
  Autoencoder} & waveform & 0.026 & 0.028 & 0.0002  &  0.0003\\
  \hline
  \makecell[l]{
  SING} & waveform &0.075 & 0.084 & 0.006 &  0.039 \\
  \hline
  \makecell[l]{
  Autoencoder} & spectral & 0.028 & 0.032 & N/A  &  N/A\\
  \hline
    \makecell[l]{
  SING} & spectral &0.039 & 0.051 & N/A  &  N/A\\
  \hline
  \makecell[l]{SING\\no time \\embedding} & spectral  & 0.050 & 0.063 & N/A  & N/A\\
  \hline
\end{tabular}
\end{center}
\caption{Results on the train and test set of the pitch completion task for different models.
The first column specifies the model, either the autoencoder used for the initial training of the LSTM
or the complete SING model with the LSTM and the convolutional decoder.
We compare models either trained with a loss on the waveform (see \eqref{eq:loss_wav}) or on the spectrograms (see \eqref{eq:loss_spec}).
Finally we also trained a model with no temporal embedding.
}
\label{table:completion_metrics}
\end{table*}

\begin{figure}
\centering
\includegraphics[width=0.2\linewidth]{images__rainbow__synth_lead_synthetic_006-045-050_gt.png}%
\includegraphics[width=0.2\linewidth]{images__rainbow__synth_lead_synthetic_006-045-050_nsynth.png}%
\includegraphics[width=0.2\linewidth]{images__rainbow__synth_lead_synthetic_006-045-050_sing.png}%
\includegraphics[width=0.2\linewidth]{images__rainbow__synth_lead_synthetic_006-045-050_wav.png}%
\includegraphics[width=0.2\linewidth]{images__rainbow__synth_lead_synthetic_006-045-050_notime.png}%
\caption{\label{figure:spectros}Example of rainbowgrams
from the NSynth dataset and the reconstructions by different models.
Rainbowgrams are defined in \cite{nsynth} as ``a CQT spectrogram with intensity
of lines proportional to the log magnitude of the power spectrum and color
given by the derivative of the phase''. Time is represented on the horizontal axis while frequencies
are on the vertical one.
From left to right:
ground truth, Wavenet-based autoencoder, SING with spectral loss, SING with waveform loss and SING without the time embedding.}
\end{figure}





\subsection{Human evaluations}
During training, we use several automatic criteria to evaluate and select our models. These criteria include the MSE on spectrograms, magnitude spectra, or waveform, and other perceptually-motivated metrics such as the Itakura-Saito divergence \cite{itakura_saito}. However, the correlation of these metrics with human perception remains imperfect, this is why we use human judgments as a metric of comparison between SING and the Wavenet baseline from \cite{nsynth}.

\subsubsection{Evaluation of perceptual quality: Mean Opinion Score}
The first characteristic that we want to measure from our generated samples is their naturalness: how good they sound to the human ear. To do so, we perform experiments on Amazon Mechanical Turk~\cite{amt} to get a Mean Opinion Score for the ground truth samples, and for the waveforms generated by SING and the Wavenet baseline. We did not include a Griffin-Lim based baseline as the authors in \cite{nsynth} concluded to the superiority of their Wavenet autoencoder.

We randomly select 100 examples from our test set. For the Wavenet-autoencoder, we pass these 100 examples through the network and retrieve the output. The latter is a pre-trained model provided by the authors of \cite{nsynth}\footnote{\url{https://github.com/tensorflow/magenta/tree/master/magenta/models/nsynth}}.
Notice that all of the 100 samples were used for training of the Wavenet-autoencoder, while they were not seen during the training of our models. For SING, we feed it the instrument, pitch and velocity information of each of the 100 samples. Workers are asked to rate the quality of the samples on a scale from 1 ("Very annoying and objectionable distortion. Totally silent audio") to 5 ("Imperceptible distortion").
Each of the 300 samples (100 samples per model) is evaluated by 60 Workers. The quality of the hardware used by Workers being variable, this could impede the interpretability of the results. Thus, we use the crowdMOS toolkit \cite{crowdmos} which detects and discards inaccurate scores. This toolkit also allows to only keep the evaluations that are made with headphones (rather than laptop speakers for example), and we choose to do so as good listening conditions are necessary to ensure the validity of our measures. We report the Mean Opinion Score for the ground-truth audio and each of the 2 models in Table \ref{tab:mos}, along with the $95\%$ confidence interval.


We observe that SING shows a significantly better MOS than the Wavenet-autoencoder baseline despite a compression factor which is 66 times higher. Moreover, to spotlight the benefits of our approach compared to the Wavenet baseline, we also report three metrics to quantify the computational load of the different models. The first metric is the training time, expressed in hours multiplied by the number of GPUs. The authors of \cite{nsynth}, mention that their model trains for 10 days on 32 GPUs, which amounts to 7680 hours*GPUs. However, the GPUs used
are capable of about half the FLOPs compared to our P100. Therefore, we corrected this value to 3840 hours*GPUs.
 On the other hand, SING is trained in 30 hours on four P100, which is 32 times faster than Wavenet. A major drawback of autoregressive models such as Wavenet is that the generation process is inherently sequential: generating the sample at time $t+1$ takes as input the sample at time $t$. We timed the generation using the implementation of the Wavenet-autoencoder provided by the authors, in its \textit{fastgen} version\footnote{\url{https://magenta.tensorflow.org/nsynth-fastgen}} which is significantly faster than the original model. This yields a 22 minutes time to generate a 4-second sample. On a single P100 GPU, Wavenet can generate up to 64 sequences at the same time before reaching the memory limit, which amounts to 0.2 seconds of audio generated per second. On the other hand, SING can generate 512 seconds of audio per second of processing time, and is thus 2500 times faster than Wavenet. Finally, SING is also efficient in memory compared to Wavenet, as the model size in MB is more than 4 times smaller than the baseline.
\begin{table}[t]
\begin{center}
\resizebox{\textwidth}{!}{
\setlength\extrarowheight{1pt}
\begin{tabular}{|c||c|c|c|c|c|} \hline
Model & MOS & Training time (hrs * GPU) & Generation speed & Compression factor & Model size \\ \hline \hline
Ground Truth  & 3.86 $\pm$ 0.24 & - & - &  - & - \\  \hline
Wavenet & 2.85 $\pm$ 0.24 & $3840^{*}$ & 0.2 sec/sec & 32 & 948 MB \\
SING & 3.55 $\pm$ 0.23 & 120 & 512 sec/sec & 2133 & 243 MB \\
\hline
\end{tabular}}
\end{center}
\caption{Mean Opinion Score (MOS) and computational load of the different models. The training time is expressed in hours * GPU units, the generation time is expressed as the number of seconds of audio that can be generated per second of processing time. The compression factor represents the ratio between the dimensionality of the audio sequences ($64,000$ values) and either the latent state of Wavenet or the input vectors to SING. We also report the size of the models, in MB.\\
$(^*)$ Time corrected to account for the difference in FLOPs of the GPUs used.}
\label{tab:mos}
\vspace*{-2ex}
\end{table}

\vspace*{-0.5ex}


\subsubsection{ABX similarity measure}
\label{sec:abx}

Besides absolute audio quality of the samples, we also want to ensure that when we condition SING on a chosen combination of instrument, pitch and velocity, we generate a relevant audio sample. To do so, we measure how close samples generated by SING are to the ground-truth relatively to the Wavenet baseline. This measure is made by performing ABX \cite{abx} experiments: the Worker is given a ground-truth sample as a reference. Then, they are presented with the corresponding samples of SING and Wavenet, in a random order to avoid bias and with the possibility of listening as many times to the samples as necessary. They are asked to pick the sample which is the closest to the reference according to their judgment. We perform this experiment on 100 ABX triplets made from the same data as for the MOS, each triplet being evaluated by 10 Workers. On average over 1000 ABX tests, $69.7\%$ are in favor of SING over Wavenet, which shows a higher similarity between our generated samples and the target musical notes than Wavenet.



  \section*{Conclusion}

  We introduced a simple model architecture, SING, based on LSTM and convolutional layers to generate waveforms.
  We achieve state-of-the-art results as measured by human evaluation on the NSynth dataset for a fraction of the training and generation cost of existing methods. We introduced a spectral loss on the generated waveform as a way of using time-frequency based metrics without requiring a post-processing step to recover the phase of a power spectrogram. We experimentally validated that SING was able to embed music notes into a small dimension vector space where the pitch, instrument and velocity were disentangled when trained with this spectral loss, as well as synthesizing pairs of instruments and pitches that were not present in the training set.
  We believe SING opens up new opportunities for lightweight quality audio synthesis with potential applications for speech synthesis and music generation.


  \subsubsection*{Acknowledgments}
  Authors thank Adam Polyak for his help on setting up the human evaluations and Axel Roebel for the insightful discussions.
  We also thank the Magenta team for their inspiring work on NSynth.



  \clearpage

  \bibliographystyle{plain}
  