Dataset Augmentation in Feature Space
Terrance DeVries and Graham W. Taylor
School of Engineering
University of Guelph
Guelph
ON N1G 2W1
Canada
\{terrance
gwtaylor\
Dataset augmentation, the practice of applying a wide array of domain-specific transformations to synthetically expand a training set, is a standard tool in supervised learning. While effective in tasks such as visual recognition, the set of transformations must be carefully designed, implemented, and tested for every new domain, limiting its re-use and generality. In this paper, we adopt a simpler, domain-agnostic approach to dataset augmentation. We start with existing data points and apply simple transformations such as adding noise, interpolating, or extrapolating between them. Our main insight is to perform the transformation not in input space, but in a learned feature space. A re-kindling of interest in unsupervised representation learning makes this technique timely and more effective. It is a simple proposal, but to-date one that has not been tested empirically. Working in the space of context vectors generated by sequence-to-sequence models, we demonstrate a technique that is effective for both static and sequential data. %Why did you do this study or project? %What did you do, and how? %What did you find? %What do your findings mean? %What are the advantages (of the method or apparatus)? %How well does it work? %Should only be one paragraph
seq2seq_paper.tex
\documentclass{article} % For LaTeX2e
\usepackage{iclr2017_workshop,times}
\usepackage{hyperref}
\usepackage{url}
\usepackage{amsmath}
\usepackage{graphicx}
\usepackage{subcaption}
\graphicspath{ {Images/} }

\title{Dataset Augmentation in Feature Space}

\author{Terrance DeVries and Graham W. Taylor\\
School of Engineering\\
University of Guelph\\
Guelph, ON N1G 2W1, Canada\\
\texttt{\{terrance,gwtaylor\}@uoguelph.ca} \\
}

% The \author macro works with any number of authors. There are two commands
% used to separate the names and addresses of multiple authors: \And and \AND.
%
% Using \And between authors leaves it to \LaTeX{} to determine where to break
% the lines. Using \AND forces a linebreak at that point. So, if \LaTeX{}
% puts 3 of 4 authors names on the first line, and the last on the second
% line, try using \AND instead of \And before the third author name.

\newcommand{\fix}{\marginpar{FIX}}
\newcommand{\new}{\marginpar{NEW}}

\begin{document}


\maketitle

\begin{abstract}
  Dataset augmentation, the practice of applying a wide array of domain-specific transformations to synthetically expand a training set, is a standard tool in supervised learning. While effective in tasks such as visual recognition, the set of transformations must be carefully designed, implemented, and tested for every new domain, limiting its re-use and generality. In this paper, we adopt a simpler, domain-agnostic approach to dataset augmentation. We start with existing data points and apply simple transformations such as adding noise, interpolating, or extrapolating between them. Our main insight is to perform the transformation not in input space, but in a learned feature space. A re-kindling of interest in unsupervised representation learning makes this technique timely and more effective. It is a simple proposal, but to-date one that has not been tested empirically. Working in the space of context vectors generated by sequence-to-sequence models, we demonstrate a technique that is effective for both static and sequential data.
%Why did you do this study or project?
%What did you do, and how?
%What did you find?
%What do your findings mean?
%What are the advantages (of the method or apparatus)?
%How well does it work?

%Should only be one paragraph


\end{abstract}

\section{Introduction}
\label{introduction}
%~1 page long
%1) X (+define X if not obvious) is an important problem
%2) The core challenges are this and that
%	Challenges listed should be things that are addresses/solved later
%3) Previous work on X has addressed these with Y, but the problems with this are Z.
%4) In this work we do W (?)
%5) This has the following appealing properties and our experiments show this and that

%Paragraph 1:
%Big Problem: not enough labeled data
%Propose Solution: Generate more data
%	1.Synthetic data can be used (Microsoft Kinect, Grand Theft Auto V), but is sometimes a poor domain match
%	2. Data augmentation - requires domain expertise to know how to transform the data

One of the major catalysts for the resurgence of neural networks as ``deep learning'' was the influx of the availability of data. Labeled data is crucial for any supervised machine learning algorithm to work, even moreso for deep architectures which are easily susceptible to overfitting. Deep learning has flourished in a few domains (e.g.~images, speech, text) where labeled data has been relatively simple to acquire. Unfortunately most of the data that is readily available is unstructured and unlabeled and this has prevented recent successes from propagating to other domains. In order to leverage the power of supervised learning, data must be manually labeled, a process which requires investment of human effort. An alternative to labeling unlabeled data is to generate new data with known labels. One variant of this approach is to create synthetic data from a simulation such as a computer graphics engine \citep{shotton2013real,Richter_2016_ECCV}, however, this may not work if the simulation is not a good representation of the real world domain. Another option is dataset augmentation, wherein the existing data is transformed in some way to create new data that appears to come from the same (conditional) data generating distribution \citep{Bengio2011-nb}. The main challenge with such an approach is that domain expertise is required to ensure that the newly generated data respects valid transformations (i.e.~those that would occur naturally in that domain).

%Paragraph 2:
%We consider augmentation in representational space
%Bengio arguments (cite paper about why feature space is better for interpolation)
%Deep learning methods can learn rich spaces through a variety of methods
%Recently there has been a return in interest to unsupervised learning techniques (VAE, GAN, GSN - generative stochastic network).

In this work, we consider augmentation not by a domain-specific transformation, but by perturbing, interpolating, or extrapolating between existing examples. However, we choose to operate not in input space, but in a learned feature space. \citet{Bengio2013} and \citet{Ozair2014} claimed that higher level representations expand the relative volume of plausible data points within the feature space, conversely shrinking the space allocated for unlikely data points. As such, when traversing along the manifold it is more likely to encounter realistic samples in feature space than compared to input space. Unsupervised representation learning models offer a convenient way of learning useful feature spaces for exploring such transformations. Recently, there has been a return to interest in such techniques, leading to, e.g.,~variational autoencoders \citep{kingma2013auto}, generative adversarial networks \citep{goodfellow2014generative}, and generative stochastic networks \citep{alain2016gsns}, each of which could be used to generate useful feature spaces for augmentation.

%Paragraph 3:
%Extrapolating/interpolating along manifold
%Compare to adding random noise
%Many different ways that you can create new data using context vectors
%We investigate some of these methods

%I think it may make sense to introduce a couple of sentences at the beginning of this paragraph stating that, although we have many options for feature learning, we adopt a seq2seq autoencoder, and modify its internal representation, called a context vector.

By manipulating the vector representation of data within a learned feature space a dataset can be augmented in a number of ways. One of the most basic transformations that can be applied to the data is to simply add random noise to the context vector.
%To increase the probability of the synthetic examples falling within the region of plausible data,
In the context of class-imbalanced data, \citet{Chawla2002} proposed \emph{interpolating} between samples in feature space. Similarly extrapolation between samples could also be applied. We investigate some of these methods to see which is most effective for improving the performance of supervised learning models when augmented data is added to the dataset.

%Paragraph 4:
%Summarize contributions of paper

%generates new data that comes from distribution of existing data
%domain independent
%actually improves performance of supervised learning algorithms (instead of just generating nice looking images)

In this work, we demonstrate that extrapolating between samples in feature space can be used to augment datasets and improve the performance of supervised learning algorithms. The main benefit of our approach is that it is domain-independent, requiring no specialized knowledge, and can therefore be applied to many different types of problems. % As opposed to most other data augmentation techniques, no domain knowledge is required to generate plausible synthetic samples.
We show that models trained on datasets that have been augmented using our technique outperform models trained only on data from the original dataset. Just as dataset augmentation in input space has become standard for visual recognition tasks, we recommend dataset augmentation in feature space as a domain-agnostic, general-purpose framework to improve generalization when limited labeled data is available.

\section{Related Work}
\label{related_work}
%~1 page long
%explore other papers that try to deal with not having enough data

%2 themed paragraphs: data augmentation and feature learning with seq2seq (mention all of the things seq2seq has worked for - translation, etc. see sequence encoder paper for reference)


%Paragraph 1: data augmentation
%EXPLORING DATA AUGMENTATION FOR IMPROVED SINGING VOICE DETECTION WITH NEURAL NETWORKS
%Alexnet
For many years, dataset augmentation has been a standard regularization technique used to reduce overfitting while training supervised learning models. Data augmentation is particularly popular for visual recognition tasks as new data can be generated very easily by applying image manipulations such as shifting, scaling, rotation, and other affine transformations. When training LeNet5, one of the most early and well-known convolutional neural network architectures, \citet{LeCun1998} applied a series of transformations to the input images in order to improve the robustness of the model. \citet{Krizhevsky2012} also used image transformations to generate new data when training the renowned AlexNet model for the 2012 Large Scale Visual Recognition Challenge (ILSVRC). They claimed that dataset augmentation reduced the error rate of the model by over 1\%. Creating new data has since been a crucial component of all recent large-scale image recognition models.

Unfortunately, dataset augmentation is not as straightforward to apply in all domains as it is for images. For example, \citet{schluter2015exploring} investigated a variety of data augmentation techniques for application to singing voice detection. These include adding Gaussian noise to the input, shifting the pitch of the audio signal, time stretching, varying the loudness of the audio signal, applying random frequency filters, and interpolating between samples in input space. They found that only pitch shifting and random frequency filtering appeared to improve model performance. While performing well on audio data, these augmentation techniques cannot be applied to other domains. As such, the process of designing, implementing, and evaluating new data augmentation techniques would need to be repeated for each new problem.

%Paragraph 2: Feature learning with seq2seq
%Sequence to Sequence Learning with Neural Networks
%Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation
%Semi-supervised Sequence Learning
%Unsupervised Learning of Video Representations using LSTMs

Important to our work are sequence-to-sequence learning (seq2seq) models which were first developed independently by \citet{cho2014learning} and \citet{sutskever2014sequence}. Generally these models convert a sequence of inputs from one domain into a fixed-length context vector which is then used to generate an output sequence, usually from a different domain. For example, the first application of seq2seq learning by Cho and Sutskever was to translate between English and French. Sequence-to-sequence learning has recently been used to achieve state-of-the-art results on a large variety of sequence learning tasks including image captioning \citep{vinyals2015show}, video captioning \citep{venugopalan2015sequence}, speech recognition (\citep{chan2015listen}, \citep{bahdanau2016end}), machine translation (\citep{cho2015using}, \citep{luong2014addressing}), text parsing \citep{vinyals2015grammar}, and conversational modeling \citep{vinyals2015neural}. The seq2seq architecture can also be used to create sequence autoencoders (SA) by creating a model that learns to reconstruct input sequences in its output \citep{srivastava2015unsupervised,dai2015semi}. We use a variant of sequence autoencoders in our work to create a feature space within which we can manipulate data to augment a training set.

\section{Model}
%1) explain clearly what is being done in the section,
%2) explain what the core challenges are
%3) explain what a baseline approach is or what others have done before
%4) motivate and explain what you do
%5) describe it.

%Setup, overall description of approach, figure of system

Our dataset augmentation technique works by first learning a data representation and then applying transformations to samples mapped to that representation. % Due to the unfolding of the manifold with each layer of representation, it is more likely that transforming and subsequently decoding a feature vector will result in the creation of valid samples, as compared to applying transformations in input space.
Our hypothesis is that, due to manifold unfolding in feature space, simple transformations applied to encoded rather than raw inputs will result in more plausible synthetic data.
While any number of representation learning models could be explored, we use a sequence autoencoder to construct a feature space. The main reason we adopt SA is that we favour a generic method that can be used for either time series or static data. %For example, images may be augmented via a PixelRNN architecture \citep[c.f.][]{van2016pixel}.
%Should add a citation for PixelRNN is mentioning it

\subsection{Sequence Autoencoder}
%Explain the sequence autoencoder used

%LSTM cells, 2 layers in encoder and decoder
%Hidden states are transferred throughout the layer stack
%Hidden states are transferred from encoder to the decoder
%Using "peek" (each time step of output gets to see context vector)
%Outputs of decoder are passed in as inputs

\begin{figure}
    \centering
    \begin{subfigure}[t]{0.308\textwidth}
        \includegraphics[width=\textwidth, trim={0 0 5.83in 0}, clip]{cropped_Model_Architecture_3.pdf}
        \caption{Sequence autoencoder}
        \label{fig:sequence_autoencoder_step}
    \end{subfigure}
    %~ %add desired spacing between images, e. g. ~, \quad, \qquad, \hfill etc.
      %(or a blank line to force the subfigure onto a new line)
    \begin{subfigure}[t]{0.429\textwidth}
        \includegraphics[width=\textwidth, trim={3.07in 0 1.54in 0}, clip]{cropped_Model_Architecture_3.pdf}
        \caption{Encode and apply data transform}
        \label{fig:data_augmentation_step}
    \end{subfigure}
    %~ %add desired spacing between images, e. g. ~, \quad, \qquad, \hfill etc.
    %(or a blank line to force the subfigure onto a new line)
    \begin{subfigure}[t]{0.153\textwidth}
        \includegraphics[width=\textwidth, trim={7.38in 0 0 0}, clip]{cropped_Model_Architecture_3.pdf}
        \caption{Decode and/or classify}
        \label{fig:decode_classify_step}
    \end{subfigure}
    \caption{System architecture composed of three steps. (a) A sequence autoencoder learns a feature space from unlabeled data, representing each sequence by a context vector ($C$). (b) Data is encoded to context vectors and augmented by adding noise, interpolating, or extrapolating (here we depict interpolation). (c) The resulting context vectors can either be used directly as features for supervised learning with a static classifier, or they can be decoded to reconstruct full sequences for training a sequence classifier.}\label{fig:system_architecture}
\end{figure}

% GWT: FYI I have modified this paragraph to make it generic to convolutional and non-convolutional autoencoders, bottleneck or overcomplete representations, and also avoid the use of ``context vector'' which I think is a seq2seq term
An autoencoder consists of two parts: an encoder and a decoder. The encoder receives data as input and, by applying one or more parametrized nonlinear transformations, converts it into a new representation, classically lower-dimensional than the original input. The decoder takes this representation and tries to reconstruct the original input, also by applying one or more nonlinear transformations. Various regularized forms of autoencoders have been proposed to learn \emph{overcomplete} representations.

A sequence autoencoder works in a similar fashion as the standard autoencoder except that the encoder and decoder use one or more recurrent layers so that they can encode and decode variable-length sequences. In all of our experiments, we use a stacked LSTM \citep{li2015constructing} with two layers for both the encoder and decoder (Figure \ref{fig:sequence_autoencoder_step}). During the forward pass, the hidden states of the recurrent layers are propagated through the layer stack. The encoder's hidden state at the final time step, called the \emph{context vector}, is used to seed the hidden state of the decoder at its first time step.\looseness=-1

The main difference between our implementation of the SA and that of \citet{dai2015semi} is how the context vector is used in the decoder. Dai and Le follow the original seq2seq approach of \citet{sutskever2014sequence} and use the context vector as input to the decoder only on the first time step, then use the output of the previous times step as inputs for all subsequent time steps as follows:
%
\begin{align*}
\mathbf{y}_{0} &= f(\mathbf{s}_{0}, \mathbf{c})\\
\mathbf{y}_{t} &= f(\mathbf{s}_{t-1}, \mathbf{y}_{t-1})
\end{align*}
%
where $f$ is the LSTM function, $\mathbf{s}$ is the state of the LSTM (both hidden and cell state), $\mathbf{c}$ is the context vector, and $\mathbf{y}$ is the output of the decoder. We instead modify the above equation so that the decoder is conditioned on the context vector at each time step as was done in \citep{cho2014learning}:
%
\begin{align*}
\mathbf{y}_{0} &= f(\mathbf{s}_{0}, \mathbf{c})\\
\mathbf{y}_{t} &= f(\mathbf{s}_{t-1}, \mathbf{y}_{t-1}, \mathbf{c}) .\
\end{align*}
%
We found that conditioning the decoder on the context vector each time step resulted in improved reconstructions, which we found to be critical to the success of the data augmentation process.

\subsection{Augmentation in Feature Space}
%Explain how context vectors are manipulated

In order to augment a dataset, each example is projected into feature space by feeding it through the sequence encoder, extracting the resulting context vector, and then applying a transformation in feature space (Figure \ref{fig:data_augmentation_step}). The simplest transform is to simply add noise to the context vectors, however, there is a possibility with this method that the resulting vector may not resemble the same class as the original, or even any of the known classes. In our experiments, we generate noise by drawing from a Gaussian distribution with zero mean and per-element standard deviation calculated across all context vectors in the dataset. We include a $\gamma$ parameter to globally scale the noise:% as shown in Equation \ref{noise_equation}.
%
\begin{equation}
\label{noise_equation}
c_{i}' = c_{i} + \gamma X,\ X \sim \mathcal{N}\{0, \sigma_{i}^{2}\}
\end{equation}
%
\noindent where $i$ indexes the elements of a context vector which corresponds to data points from the training set. A more directed approach for data augmentation follows the techniques introduced by \citet{Chawla2002}. For each sample in the dataset, we find its $K$ nearest neighbours in feature space which share its class label. For each pair of neighbouring context vectors, a new context vector can then be generated using interpolation:% as shown in Equation \ref{interpolation_equation}:
%
\begin{equation}
\label{interpolation_equation}
\mathbf{c}' = (\mathbf{c}_{k} - \mathbf{c}_{j}) \lambda\ + \mathbf{c}_{j}
\end{equation}

where $\mathbf{c}'$ is the synthetic context vector, $\mathbf{c}_{i}$ and $\mathbf{c}_{j}$ are neighbouring context vectors, and $\lambda$ is a variable in the range $\{0,1\}$ that controls the degree of interpolation. In our experiments, we use $\lambda = 0.5$ so that the new sample balances properties of both original samples. In a similar fashion, extrapolation can also be applied to the context vectors:% as shown by Equation \ref{extrapolation_equation}:
%
\begin{equation}
\label{extrapolation_equation}
\mathbf{c}_{j}' = (\mathbf{c}_{j} - \mathbf{c}_{k}) \lambda\ + \mathbf{c}_{j} .\
\end{equation}

In the case of extrapolation, $\lambda$ is a value in the range $\{0, \infty\}$ which controls the degree of extrapolation. While $\lambda$ could be drawn from a random distribution for each new sample we found that setting $\lambda = 0.5$ worked well as a default value in most cases, so we use this setting in all of our tests.
%We found that setting $\lambda = 0.5$ produced good results and use this in all of our tests.

Once new context vectors have been created, they can either be used directly as input for a learning task, or they can be decoded to generate new sequences (Figure \ref{fig:decode_classify_step}). When interpolating between two samples, the resulting decoded sequence is set to be the average length of the two inputs. When extrapolating between two samples the length of the new sequence is set to be the same as that of $c_{j}$.

\section{Experiments}
%One subsection per dataset
%Generic intro to experimental setup
%In each section explain dataset, any modifications made to general model, and then results

%Small amount of dropout
%Trained using ADAM
%Masking used for variable length sequences
%Reverse inputs

In all experiments, we trained a LSTM-based sequence autoencoder in order to learn a feature space from the available training examples. Each hidden layer, including the context vector, had the same number of hidden units and a dropout probability of $p=0.2$. The autoencoders were trained using Adam \citep{kingma2014adam} with an initial learning rate of 0.001, which was reduced by half whenever no improvement was observed in the validation set for 10 epochs. Finally, we reversed the order of the input sequences as suggested by \citet{sutskever2014sequence}. We found that reversing the order of input sequences caused the model to train faster and achieve better final solutions.

For all classification experiments where interpolation or extrapolation was applied to generate new samples, we applied the following procedure unless otherwise stated. For each sample in the dataset we found the 10 nearest in-class neighbours by searching in feature space. We then interpolated or extrapolated between each neighbour and the original sample to produce a synthetic example which was added to the augmented dataset. For all tests, the baseline model and the augmented dataset model(s) were trained for the same number of weight updates regardless of dataset size.

\subsection{Visualization - Sinusoids}
\label{subsection:sinusoids}

To gain an intuition of the method we start by working with a synthetic dataset of sinusoids. Sinusoids work well as a test case for this technique as they have a known behaviour and only two dimensions (amplitude and time), so we can easily observe the effects of the dataset augmentation process. To create a training set, sinusoids were generated with amplitude, frequency, and phase drawn from a uniform distribution.

For this toy problem, we trained a sequence autoencoder with 32 hidden units in each layer. We then applied different data augmentation strategies to observe the effects on the ``synthetic'' sinusoids. For each test we extracted the context vectors of two input sinusoids, performed an operation, then decoded the resulting context vectors to generate new sequences.

We first augmented data by adding random noise to the context vectors before decoding. The noise magnitude parameter $\gamma$ from Equation \ref{noise_equation} was set to 0.5. In Figure \ref{fig:random_sinusoid} the blue and green ``parent'' samples are shown in bold while the augmented ``child'' samples are thinner, lighter lines. Importantly, we observe that all new samples are valid sinusoids with stable, regular repeating patterns. Although mimicking the major properties of their parents the generated samples have small changes in amplitude, frequency, and phase, as would be the expected effect for the addition of random noise. %I feel like this should have a conclusion at then end

\begin{figure}
    \centering
    \begin{subfigure}[t]{0.3\textwidth}
        \fbox{\includegraphics[width=\textwidth, trim={1cm 0 8in 0}, clip]{random_no_border.pdf}}
        \caption{Random noise}
        \label{fig:random_sinusoid}
    \end{subfigure}
    \quad %add desired spacing between images, e. g. ~, \quad, \qquad, \hfill etc.
      %(or a blank line to force the subfigure onto a new line)
    \begin{subfigure}[t]{0.3\textwidth}
        \fbox{\includegraphics[width=\textwidth, trim={4.2in 0 4.2in 0}, clip]{interpolate_no_border.pdf}}
        \caption{Interpolation}
        \label{fig:interp_sinusoid}
    \end{subfigure}
    \quad %add desired spacing between images, e. g. ~, \quad, \qquad, \hfill etc.
    %(or a blank line to force the subfigure onto a new line)
    \begin{subfigure}[t]{0.3\textwidth}
         \fbox{\includegraphics[width=\textwidth, trim={8in 0 1cm 0}, clip]{extrapolate_no_border.pdf}}
        \caption{Extrapolation}
        \label{fig:extrap_sinusoid}
    \end{subfigure}
    \caption{Sinusoids with various transformations applied in feature space. (a) Random noise added with $\gamma = 0.5$. (b) Interpolation between two sinusoids for values of $\lambda$ between 0 and 1. (c) Extrapolation between two sinusoids for values of $\lambda$ between 0 and 1. Best viewed in colour.}\label{fig:sinusoids}
\end{figure}

For a more directed form of data augmentation we experimented with interpolating between sinusoids within the space of the context vectors. Figure \ref{fig:interp_sinusoid} demonstrates interpolation between two sinusoids using Equation \ref{interpolation_equation} while varying the $\lambda$ parameter from 0 to 1. Unlike the results obtained by \citet{Bengio2013} where the transition between classes occurs very suddenly we find that the samples generated by our model smoothly transition between the two parent sinusoids. This is an exciting observation as it suggests that we can control characteristics of the generated samples by combining two samples which contain the desired properties.

In a similar fashion to interpolation we can also extrapolate between two samples using Equation \ref{extrapolation_equation}. For this experiment we again vary the $\lambda$ parameter from 0 to 1 to generate a range of samples. As seen in Figure \ref{fig:extrap_sinusoid}, this appears to have the effect of exaggerating properties of each sinusoid with respect to the properties of the other sinusoid. For example, we see that new samples generated from the blue parent sinusoid increase in amplitude and decrease in phase shift. Conversely, samples generated from the green parent sinusoid decrease in amplitude and increase in phase shift. The behaviour of the extrapolation operation could prove very beneficial for data augmentation as it could be used to generate extra samples of rare or underrepresented cases within the dataset, which is a common failure case.

\subsection{Visualization - UJI Pen Characters}

The UJI Pen Characters dataset (v2)  contains 11,640 instances of 97 different characters hand-written by 60 participants \citep{llorens2008ujipenchars}. All samples were collected using a tablet PC and a stylus. Characters are defined by a sequence of X and Y coordinates, and include upper and lower case ASCII letters, Spanish non-ASCII letters, the 10 digits, and other common punctuation and symbols. As with the sinusoids in Section \ref{subsection:sinusoids}, handwritten characters are suitable for evaluating dataset augmentation methods as they have an expected shape and can be easily visualized.

As a preprocessing step for this dataset we first applied local normalization to each sample to get a fixed size, followed by a global normalization across the dataset as a whole. A sequence autoencoder with 128 hidden units per layer was trained to construct the feature space within which data augmentation could take place. Figure \ref{fig:interp_chars} demonstrates the effects of interpolating between characters in feature space. In this example we use the ``@'' symbol. We see that the resulting characters share characteristics of the two parent inputs, such as the length of the symbol's tail or the shape of the central ``a''. Visually the majority of generated samples appear very similar to their parents, which is expected from interpolation, but is not necessarily useful from the perspective of data augmentation.

% \begin{figure}[h]
%     \centering
%     \includegraphics[width=\textwidth, trim={0 0.8cm 0 0.8cm}, clip]{interp_char_table.pdf}
%     \caption{Interpolation between handwritten characters. Character (0,i) is interpolated with character (j,0) to form character (i,j), where i is the row number and j is the column number. Original characters are shown in bold.}
%     \label{fig:interp_chars}
% \end{figure}

When augmenting data for the purpose of improving performance of machine learning algorithms it is desirable to create samples that are different from the data that is already common in the dataset. To this end, extrapolating between samples is preferable, as shown in Figure \ref{fig:exterp_chars}. Extrapolated data displays a wider variety compared to samples created by interpolation. We hypothesize that it is this added variability that is necessary in order for data augmentation to be useful. %Is this necessary to state? It seems like an established fact already.

% \begin{figure}[h]
%     \centering
%     \includegraphics[width=\textwidth, trim={0 0.8cm 0 0.8cm}, clip]{exterp_char_table.pdf}
%     \caption{Extrapolation between handwritten characters. Character (1,i) is extrapolated with character (j,1) to form character (i,j), where i is the row number and j is the column number. Original characters are shown in bold.}
%     \label{fig:exterp_chars}
% \end{figure}

\begin{figure}
    \centering
    \begin{subfigure}[t]{0.48\textwidth}
        \fbox{\includegraphics[width=\textwidth, trim={1.3in 0.5cm 4.9in 0.5cm}, clip]{interp_char_table.pdf}}
        \caption{Interpolation}
        \label{fig:interp_chars}
    \end{subfigure}
    \quad %add desired spacing between images, e. g. ~, \quad, \qquad, \hfill etc.
      %(or a blank line to force the subfigure onto a new line)
    \begin{subfigure}[t]{0.48\textwidth}
        \fbox{\includegraphics[width=\textwidth, trim={1.3in 0.5cm 4.9in 0.5cm}, clip]{exterp_char_table.pdf}}
        \caption{Extrapolation}
        \label{fig:exterp_chars}
    \end{subfigure}
    \caption{Interpolation (a) and extrapolation (b) between handwritten characters. Character (0,i) is interpolated/extrapolated with character (j,0) to form character (i,j), where i is the row number and j is the column number. Original characters are shown in bold.}\label{fig:ujichars}
\end{figure}

\subsection{Spoken Arabic Digits}
\label{subsection:arabicDigits}

For our first quantitative test we use the Arabic Digits dataset \citep{Lichman:2013} which contains 8,800 samples of time series mel-frequency cepstrum coefficients (MFCCs) extracted from audio clips of spoken Arabic digits. Thirteen MFCCs are available for each time step in this dataset. To preprocess the data we apply global normalization. To evaluate our data augmentation techniques we used the official train/test split and trained ten models with different random weight initializations.

As a baseline model we trained a simple two layer MLP on the context vectors produced by a SA. Both models used 256 hidden units in each hidden layer. The MLP applied dropout with $p=0.5$ after each dense layer. To evaluate the usefulness of different data augmentation techniques we trained a new baseline model on datasets that had been augmented with newly created samples. The techniques we evaluated were: adding random noise to context vectors, interpolating between two random context vectors from the same class, interpolating between context vectors and their nearest neighbours from the same class, and extrapolating between context vectors and their nearest neighbours from the same class. The results of our tests are summarized in Table \ref{arabicDigitsErrorTable}.

\begin{table}[h]
\centering
\caption{Test set error on Arabic Digits dataset averaged over 10 runs}
\label{arabicDigitsErrorTable}
\begin{tabular}{ll}
\textbf{Model} & \textbf{Test Error (\%)} \\ \hline
Baseline (ours)      & $1.36 \pm 0.15$      \\
Baseline + random noise (ours) & $1.10 \pm 0.15$ \\
Baseline + random interpolation (ours) & $1.82 \pm 0.21$ \\ %interpolation using a random sample from the same class
Baseline + nearest neighbour interpolation (ours) & $1.57 \pm 0.19$ \\ %interpolation using 10 nearest neighbours
Baseline + nearest neighbour extrapolation (ours)  & $\mathbf{0.74 \pm 0.11}$      \\
\citep{hammami2012spoken} & $\mathbf{0.69}$ %Hammami uses GMM
\end{tabular}
\end{table}

We find that our simple baseline model achieves competitive performance after training on the extracted context vectors, demonstrating the feature extracting capability of the sequence autoencoder. The na\"ive data augmentation approach of adding random noise to the context vectors further improves performance. Of interest, we find that adding new samples generated using interpolation techniques diminishes the performance of the model, which confirms our hypothesis that good data augmentation techniques should add variability to the dataset. % Interpolation only repeats what is already existing within the dataset.
%Perhaps the claim about good data augmentation techniques needs should be stronger?
Of the two interpolation techniques, we see that interpolating between neighbouring samples performs better than simply interpolating with randomly chosen samples of the same class. %Some comment about nearest neighbours being critical to success would be good. Link to Chawla, 2002?
Finally we observe that extrapolating between samples improves model performance significantly, reducing the baseline error rate by almost half. Our results rival those of \citet{hammami2012spoken}, which to our knowledge are state-of-the-art on this dataset.\looseness=-1


\subsection{Australian Sign Language Signs (AUSLAN)}

Our second quantitative test was conducted on the Australian Sign Language Signs dataset (AUSLAN). AUSLAN was produced by \citet{kadous2002temporal} and contains 2,565 samples of a native signer signing 95 different words or phrases while wearing high quality position tracking gloves. Each time series sample is, on average, 57 frames in length and includes 22 features: roll, pitch, yaw, finger bend, and the 3D coordinates of each hand. To preprocess the raw data we first locally centre each sample and then apply global normalization. For evaluation, we perform cross validation with 5 folds, as is common practice for the AUSLAN dataset.

The baseline model for these tests was a two layer MLP with 512 hidden units in each layer, with dropout ($p=0.5$) applied on each. Similar to Arabic Digits, dataset we find that the simple MLP can achieve competitive results when trained on the context vectors extracted from the sequence autoencoder (see Table \ref{auslanError}). In this case, however, we observe that adding random noise to the context vectors did not improve performance. One possible explanation for this outcome is that the AUSLAN dataset has much more classes than the Arabic Digits dataset (95 versus 10) so there is higher probability of a randomly augmented context vector jumping from one class manifold to another. Traversing instead along the representational manifold in a directed manner by extrapolating between neighbouring samples results in improved performance over that of the baseline model. Our results also match the performance of \citet{rodriguez2005support}, which to our knowledge is the best 5-fold cross validation result for the AUSLAN dataset.

\begin{table}[h]
\centering
\caption{CV error on AUSLAN dataset averaged over 5 folds}
\label{auslanError}
\begin{tabular}{ll}
\textbf{Model} & \textbf{Test Error (\%)} \\ \hline
Baseline (ours)      & $1.53 \pm 0.26$      \\
Baseline + random noise (ours) & $1.67 \pm 0.12$ \\
Baseline + interpolation (ours) & $1.87 \pm 0.44$ \\ %interpolation using 10 nearest neighbours
Baseline + extrapolation (ours)  & $\mathbf{1.21 \pm 0.26}$      \\
\citep{rodriguez2005support} & $\mathbf{1.28}$
\end{tabular}
\end{table}

\subsection{UCFKinect}

The final time series dataset we considered was the UCF Kinect action recognition dataset \citep{ellis2013exploring}. It contains motion capture data of participants performing 16 different actions such as \textit{run, kick, punch,} and \textit{hop}. The motion capture data consists of 3-dimensional coordinates for 15 skeleton joints for a total of 45 attributes per frame. In total there are 1,280 samples within the dataset. To preprocess the dataset we first shift the coordinates of each sample so that the central shoulder joint of the first frame is located at the origin. Global normalization is also applied.

%Our secondary goal with the UCFKinect dataset was to quantitatively evaluate the performance of extrapolation-based data augmentation.
To compare to previous results, we used 4-fold cross validation (see Table \ref{table:ucfKinectAccuracy} for a summary of results). We found that extrapolating between samples in representational space improved the performance of our untuned model by more than 1\%, which is quite significant. Our results are 2.5 percentage points below the current state-of-the-art result produced by \citet{beh2014hidden}, but further tuning of the model could improve results.

\begin{table}[h]
\centering
\caption{CV error on UCFKinect dataset averaged over 4 folds}
\label{table:ucfKinectAccuracy}
\begin{tabular}{ll}
\textbf{Model} & \textbf{Test Error (\%)} \\ \hline
Baseline (ours)       & $4.92 \pm 2.09$     \\
Baseline + extrapolation (ours)  & $3.59 \pm 1.61$     \\
\citep{beh2014hidden} & $\mathbf{1.10}$
\end{tabular}
\end{table}

\subsection{Image Classification: MNIST and CIFAR-10}
\label{MNISTandCIFAR10}

% Subsection: Image Classification: MNIST and CIFAR-10
% - P1 The point of these experiments is to demonstrate that our technique works for static data; it works in a domain where data augmentation is prevalent; and it outperforms traditional augmentation in input space (?)

Having successfully applied dataset augmentation in feature space to improve the accuracy of sequence classification tasks, we now experiment with applying our technique to static data. For these experiments we concentrate on the image domain where manual data augmentation is already prevalent. We find that augmenting datasets by extrapolating within a learned feature space improves classification accuracy compared to no data augmentation, and in some cases surpasses traditional (manual) augmentation in input space.

% - P2 Sentence to introduce MNIST; it consists of 28 x 28 grayscale images 60,000 train, 10,000 test. Sentence to introduce CIFAR-10; ...

In our experiments we consider two commonly used small-scale image datasets: MNIST and CIFAR-10. MNIST consists of 28$\times$28 greyscale images containing handwritten digits from 0 to 9. There are 60,000 training images and 10,000 test images in the official split. CIFAR-10 consists of 32$\times$32 colour images containing objects in ten generic object categories. This dataset is typically split into 50,000 training and 10,000 test images.

% - P3 Brief paragraph about the architecture. The *same* architecture was used for MNIST/CIFAR experiments. We used the seq2seq architecture proposed by Dai and Le (2015) who also considered static image data under the seq2seq framework. Any specifics about pre-processing for CIFAR (adopt pipeline from [X]). Any specifics about preprocessing for MNIST.

In all of our image experiments, we apply the same sequence autoencoder (SA) architecture as shown in Figure \ref{fig:sequence_autoencoder_step} to learn a representation. No pre-processing beyond a global scaling is applied to the MNIST dataset. For CIFAR-10 we apply global normalization and the same crop and flip operations that \citet{Krizhevsky2012} used for input space data augmentation when training AlexNet (we crop to 24$\times$24). To simulate sequence input the images are fed into the network one row of pixels per time step similar to the SA setup in \citep{dai2015semi}.

% - P4 Explain experimental scenario. For both datasets, we considered the same set of tests: (1), (2), (3), (4) as discussed last night.  Results are shown in Table X and Table Y. Explain main findings.

For each dataset we train a 2-layer MLP on the context vectors produced by the sequence encoder. Both MLP and SA use the same number of hidden units in each layer: 256 per layer for MNIST and 1024 per layer for CIFAR-10. We conduct four different test scenarios on the MNIST dataset. To control for the representation, as a baseline we trained the classifier only on context vectors from the original images (i.e.~SA with no augmentation). We then compare this to training with various kinds of dataset augmentation: traditional affine image transformations in input space (shifting, rotation, scaling), extrapolation between nearest neighbours in input space, and extrapolation between nearest neighbours in representational space. For both extrapolation experiments we use three nearest neighbours per sample and $\gamma = 0.5$ when generating new data. For CIFAR-10, our baseline is trained using context vectors extracted from cropped and flipped images. Against this baseline we test the addition of extrapolation between nearest neighbours in representational space, using the same setup as the MNIST test. Due to the size of the datasets we apply an approximate nearest neighbour algorithm \citep{wan2015hdidx}.

Results are reported in Table \ref{mnistcifar}. For MNIST, we find that extrapolating in feature space not only performs better than the baseline, but it also achieves a lower error rate compared to domain-specific data augmentation in input space. A similar outcome is observed in CIFAR-10, where feature space extrapolation reduces the error rate by 1.4\%. Interestingly, we note that the baseline test for this dataset already leveraged image transformations to improve performance, so the additional reduction in error rate could indicate that both kinds of augmentation, extrapolation in feature space and manual transformation in pixel space, could complement each other.

\begin{table}[h]
\centering
\caption{Test error (\%) on MNIST and CIFAR-10. Averaged over 10 runs.}
\label{mnistcifar}
\begin{tabular}{lll}
\textbf{Model}                         & MNIST & CIFAR-10 \\ \hline
Baseline                               & $1.093 \pm 0.057$ & $30.65 \pm 0.27$    \\
Baseline + input space affine transformations    & $1.477 \pm 0.068$ & \multicolumn{1}{c}{-}   \\
Baseline + input space extrapolation   & $1.010 \pm 0.065$  & \multicolumn{1}{c}{-}   \\
Baseline + feature space extrapolation & $\mathbf{0.950 \pm 0.036}$ & $\mathbf{29.24 \pm 0.27}$
\end{tabular}
\end{table}

\subsubsection{Classification on Reconstructed Images}

In order to determine if performance gains are retained when using reconstructed images instead of context vectors we conducted additional tests using the Wide ResNet architecture introduced by \cite{zagoruyko2016wide}. For our implementation of Wide ResNet we use a depth factor of 28 and a widening factor of $k = 4$. Experiments were conducted on the original, unmodified dataset (Test 1), 24$\times$24 central crops (Test 2), central crops with extrapolation (Test 3), 24$\times$24 patches with simple data augmentation (shifting and mirroring) (Test 4), and 24$\times$24 patches with simple data augmentation and extrapolation (Test 5).

When augmenting data in feature space for these experiments we follow the same procedure that was used for our previous CIFAR-10 setup in Section \ref{MNISTandCIFAR10}, with the addition that extrapolated context vectors are decoded to return samples into image space. We ran two different sets of tests for all test scenarios. For the first set of tests we use the original images, with image reconstructions of extrapolated samples added to the dataset for Tests 3 and 5. In the second set of tests, we also encode and decode all original images with the sequence autoencoder so that they share the reconstruction properties of the extrapolated examples. The results of both sets of testing are shown in Table \ref{table:wide_resnet}.

\begin{table}[h]
\centering
\caption{Test error (\%) on CIFAR-10 with Wide ResNet architecture. Average over 10 runs. Note that we do not perform a reconstruction test on 32$\times$32 as the sequence autoencoder was trained on 24$\times$24 images.}
\label{table:wide_resnet}
\begin{tabular}{cclcc}
\textbf{\begin{tabular}[c]{@{}c@{}}\\ Test \\ ID\end{tabular}} & \textbf{\begin{tabular}[c]{@{}c@{}}\\ Image\\ Size\end{tabular}} & \textbf{\begin{tabular}[c]{@{}l@{}}\\ \\ Description\end{tabular}} & \textbf{\begin{tabular}[c]{@{}c@{}}\\ \\ Test Error\end{tabular}} & \textbf{\begin{tabular}[c]{@{}c@{}}Test Error\\ (Reconstructions\\ of original data)\end{tabular}} \\ \hline
1 & 32$\times$32 & Original dataset            & $8.59 \pm 0.24$          & - \\
2 & 24$\times$24 & Center crop                 & $11.28 \pm 0.25$         & $18.54 \pm 0.38$ \\
3 & 24$\times$24 & Center crop + extrapolation & $13.90 \pm 0.22$         & $17.69 \pm 0.39$ \\
4 & 24$\times$24 & Simple data augmentation    & $\mathbf{7.33 \pm 0.17}$ & $13.60 \pm 0.17$ \\
5 & 24$\times$24 & Simple data augmentation + extrapolation   & $8.80 \pm 0.24$          & $\mathbf{12.00 \pm 0.23}$
\end{tabular}
\end{table}

While overall performance improved compared to training on context vectors, we find that the addition of extrapolated image reconstructions actually worsened performance when used in conjunction with original images. However, we also observed that networks trained entirely with reconstructed examples performed considerably worse compared to training on unmodified original examples. We believe that this discrepancy indicates a significant loss of fidelity of the examples (both original and extrapolated) when mapping to and from context vectors, which could explain the reduction in accuracy. When training networks entirely with reconstructed samples we observe that the addition of extrapolated samples improves performance: by 1.0\% when training only on the center crop of images, and by 1.6\% when applied alongside input space data augmentation. These results mirror what we observed in our experiments in Section \ref{MNISTandCIFAR10}, again suggesting that feature space extrapolation and domain-specific input space augmentation could be complimentary.

\section{Discussion}
\label{discussion}
%Additional synthetic data experiments where we can control the complexity of the decision boundary reveal
An important finding is that the extrapolation operator, when used in feature space, generated useful synthetic examples while noise and interpolation did not. Additional synthetic data experiments where we could control the complexity of the decision boundary revealed that extrapolation only improved model performance in cases where there were complex class boundaries. In cases with simple class boundaries, such as linear separability or one class encircling another, extrapolation hindered model performance, while interpolation helped. Our current hypothesis is that interpolation tends to tighten class boundaries and unnecessarily increase confidence, leading to overfitting. This behaviour may cause the model to ignore informative extremities that can describe a complex decision boundary and as a result produce an unnecessarily smooth decision boundary. As most high-dimensional, real datasets will typically have complex decision boundaries, we find extrapolation to be well suited for dataset augmentation in feature space.

\section{Conclusion}
\label{conclusion}
%trend appears to be that it works better with more data
%would probably work well for semi-supervised learning
In this paper, we demonstrate a new domain-independent data augmentation technique that can be used to improve performance when training supervised learning models. We train a sequence autoencoder to construct a learned feature space in which we extrapolate between samples. This technique allows us to increase the amount of variability within the dataset, ultimately resulting in a more robust model. We demonstrate our technique quantitatively on five datasets from different domains (speech, sensor processing, motion capture, and images) \emph{using the same simple architecture} and achieve near state-of-the-art results on two of them. Moreover, employing the technique in conjunction with more sophisticated domain-specific architectures (Wide ResNets), we show that data augmentation in feature space may complement domain-specific augmentation.

%that extrapolation in representational space can be used to generate new samples that are useful for data augmentation
%achieve close to state-of-the-art performance in two datasets
%demonstrate that our technique is domain independent, and can therefore be applied to many different problems


\bibliography{iclr2017_workshop}
\bibliographystyle{iclr2017_workshop}

\end{document}

seq2seq_paper.bbl
\begin{thebibliography}{35}
\providecommand{\natexlab}[1]{#1}
\providecommand{\url}[1]{\texttt{#1}}
\expandafter\ifx\csname urlstyle\endcsname\relax
  \providecommand{\doi}[1]{doi: #1}\else
  \providecommand{\doi}{doi: \begingroup \urlstyle{rm}\Url}\fi

\bibitem[Alain et~al.(2016)Alain, Bengio, Yao, Yosinski, Thibodeau-Laufer,
  Zhang, and Vincent]{alain2016gsns}
Guillaume Alain, Yoshua Bengio, Li~Yao, Jason Yosinski, {\'E}ric
  Thibodeau-Laufer, Saizheng Zhang, and Pascal Vincent.
\newblock {GSNs}: generative stochastic networks.
\newblock \emph{Information and Inference}, 5\penalty0 (2):\penalty0 210--249,
  2016.

\bibitem[Bahdanau et~al.(2016)Bahdanau, Chorowski, Serdyuk, Bengio,
  et~al.]{bahdanau2016end}
Dzmitry Bahdanau, Jan Chorowski, Dmitriy Serdyuk, Yoshua Bengio, et~al.
\newblock End-to-end attention-based large vocabulary speech recognition.
\newblock In \emph{2016 IEEE International Conference on Acoustics, Speech and
  Signal Processing (ICASSP)}, pp.\  4945--4949. IEEE, 2016.

\bibitem[Beh et~al.(2014)Beh, Han, Durasiwami, and Ko]{beh2014hidden}
Jounghoon Beh, David~K Han, Ramani Durasiwami, and Hanseok Ko.
\newblock Hidden {Markov} model on a unit hypersphere space for gesture
  trajectory recognition.
\newblock \emph{Pattern Recognition Letters}, 36:\penalty0 144--153, 2014.

\bibitem[Bengio et~al.(2011)Bengio, Bastien, Bergeron, Boulanger-Lewandowski,
  Breuel, Chherawala, Cisse, C\^{o}t\'{e}, Erhan, Eustache, and
  {Others}]{Bengio2011-nb}
Yoshua Bengio, Fr\'{e}d\'{e}ric Bastien, Arnaud Bergeron, Nicolas
  Boulanger-Lewandowski, Thomas~M Breuel, Youssouf Chherawala, Moustapha Cisse,
  Myriam C\^{o}t\'{e}, Dumitru Erhan, Jeremy Eustache, and {Others}.
\newblock Deep learners benefit more from {Out-of-Distribution} examples.
\newblock In \emph{International Conference on Artificial Intelligence and
  Statistics (AISTATS)}, pp.\  164--172, 2011.

\bibitem[Bengio et~al.(2013)Bengio, Mesnil, Dauphin, and Rifai]{Bengio2013}
Yoshua Bengio, Gr{\'e}goire Mesnil, Yann Dauphin, and Salah Rifai.
\newblock Better mixing via deep representations.
\newblock In \emph{ICML (1)}, pp.\  552--560, 2013.

\bibitem[Chan et~al.(2016)Chan, Jaitly, Le, and Vinyals]{chan2015listen}
William Chan, Navdeep Jaitly, Quoc~V Le, and Oriol Vinyals.
\newblock Listen, attend and spell: A neural network for large vocabulary
  conversational speech recognition.
\newblock In \emph{2016 IEEE International Conference on Acoustics, Speech and
  Signal Processing (ICASSP)}, pp.\  4960--4964. IEEE, 2016.

\bibitem[Chawla et~al.(2002)Chawla, Bowyer, Hall, and Kegelmeyer]{Chawla2002}
Nitesh~V. Chawla, Kevin~W. Bowyer, Lawrence~O. Hall, and W.~Philip Kegelmeyer.
\newblock Smote: synthetic minority over-sampling technique.
\newblock \emph{Journal of Artificial Intelligence Research}, 16:\penalty0
  321--357, 2002.

\bibitem[Cho et~al.(2014)Cho, van Merri{\"{e}}nboer, G{\"{u}}l{\c c}ehre,
  Bahdanau, Bougares, Schwenk, and Bengio]{cho2014learning}
Kyunghyun Cho, Bart van Merri{\"{e}}nboer, {\c C}ağlar G{\"{u}}l{\c c}ehre,
  Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio.
\newblock Learning phrase representations using rnn encoder--decoder for
  statistical machine translation.
\newblock In \emph{Proceedings of the 2014 Conference on Empirical Methods in
  Natural Language Processing (EMNLP)}, pp.\  1724--1734, 2014.

\bibitem[Dai \& Le(2015)Dai and Le]{dai2015semi}
Andrew~M Dai and Quoc~V Le.
\newblock Semi-supervised sequence learning.
\newblock In \emph{Advances in Neural Information Processing Systems}, pp.\
  3079--3087, 2015.

\bibitem[Ellis et~al.(2013)Ellis, Masood, Tappen, Laviola~Jr, and
  Sukthankar]{ellis2013exploring}
Chris Ellis, Syed~Zain Masood, Marshall~F Tappen, Joseph~J Laviola~Jr, and
  Rahul Sukthankar.
\newblock Exploring the trade-off between accuracy and observational latency in
  action recognition.
\newblock \emph{International Journal of Computer Vision}, 101\penalty0
  (3):\penalty0 420--436, 2013.

\bibitem[Goodfellow et~al.(2014)Goodfellow, Pouget-Abadie, Mirza, Xu,
  Warde-Farley, Ozair, Courville, and Bengio]{goodfellow2014generative}
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley,
  Sherjil Ozair, Aaron Courville, and Yoshua Bengio.
\newblock Generative adversarial nets.
\newblock In \emph{Advances in Neural Information Processing Systems}, pp.\
  2672--2680, 2014.

\bibitem[Hammami et~al.(2012)Hammami, Bedda, and Farah]{hammami2012spoken}
Nacereddine Hammami, Mouldi Bedda, and Nadir Farah.
\newblock Spoken {Arabic} digits recognition using {MFCC} based on {GMM}.
\newblock In \emph{Sustainable Utilization and Development in Engineering and
  Technology (STUDENT), 2012 IEEE Conference on}, pp.\  160--163. IEEE, 2012.

\bibitem[Jean et~al.(2015)Jean, Cho, Memisevic, and Bengio]{cho2015using}
S{\'{e}}bastien Jean, Kyunghyun Cho, Roland Memisevic, and Yoshua Bengio.
\newblock On using very large target vocabulary for neural machine translation.
\newblock In \emph{Proceedings of the 53rd Annual Meeting of the Association
  for Computational Linguistics and the 7th International Joint Conference on
  Natural Language Processing}, pp.\  1--10, 2015.

\bibitem[Kadous(2002)]{kadous2002temporal}
Mohammed~Waleed Kadous.
\newblock \emph{Temporal classification: Extending the classification paradigm
  to multivariate time series}.
\newblock PhD thesis, The University of New South Wales, 2002.

\bibitem[Kingma \& Ba(2015)Kingma and Ba]{kingma2014adam}
Diederik Kingma and Jimmy Ba.
\newblock Adam: A method for stochastic optimization.
\newblock In \emph{The International Conference on Learning Representations
  (ICLR)}, 2015.

\bibitem[Kingma \& Welling(2014)Kingma and Welling]{kingma2013auto}
Diederik Kingma and Max Welling.
\newblock Auto-encoding variational bayes.
\newblock In \emph{The International Conference on Learning Representations
  (ICLR)}, 2014.

\bibitem[Krizhevsky et~al.(2012)Krizhevsky, Sutskever, and
  Hinton]{Krizhevsky2012}
Alex Krizhevsky, Ilya Sutskever, and Geoffrey~E Hinton.
\newblock Imagenet classification with deep convolutional neural networks.
\newblock In \emph{Advances in neural information processing systems}, pp.\
  1097--1105, 2012.

\bibitem[LeCun et~al.(1998)LeCun, Bottou, Bengio, and Haffner]{LeCun1998}
Yann LeCun, L{\'e}on Bottou, Yoshua Bengio, and Patrick Haffner.
\newblock Gradient-based learning applied to document recognition.
\newblock \emph{Proceedings of the IEEE}, 86\penalty0 (11):\penalty0
  2278--2324, 1998.

\bibitem[Li \& Wu(2015)Li and Wu]{li2015constructing}
Xiangang Li and Xihong Wu.
\newblock Constructing long short-term memory based deep recurrent neural
  networks for large vocabulary speech recognition.
\newblock In \emph{2015 IEEE International Conference on Acoustics, Speech and
  Signal Processing (ICASSP)}, pp.\  4520--4524. IEEE, 2015.

\bibitem[Lichman(2013)]{Lichman:2013}
Moshe Lichman.
\newblock {UCI} machine learning repository, 2013.
\newblock URL \url{http://archive.ics.uci.edu/ml}.

\bibitem[Llorens et~al.(2008)Llorens, Prat, Marzal, Vilar, Castro, Amengual,
  Barrachina, Castellanos, Boquera, G{\'o}mez, et~al.]{llorens2008ujipenchars}
David Llorens, Federico Prat, Andr{\'e}s Marzal, Juan~Miguel Vilar,
  Mar{\'\i}a~Jos{\'e} Castro, Juan-Carlos Amengual, Sergio Barrachina, Antonio
  Castellanos, Salvador~Espa{\~n}a Boquera, JA~G{\'o}mez, et~al.
\newblock The {UJI}penchars database: a pen-based database of isolated
  handwritten characters.
\newblock In \emph{LREC}, 2008.

\bibitem[Luong et~al.(2015)Luong, Sutskever, Le, Vinyals, and
  Zaremba]{luong2014addressing}
Minh-Thang Luong, Ilya Sutskever, Quoc~V Le, Oriol Vinyals, and Wojciech
  Zaremba.
\newblock Addressing the rare word problem in neural machine translation.
\newblock In \emph{Proceedings of the 53rd Annual Meeting of the Association
  for Computational Linguistics and the 7th International Joint Conference on
  Natural Language Processing}, pp.\  11--19, 2015.

\bibitem[Ozair \& Bengio(2014)Ozair and Bengio]{Ozair2014}
Sherjil Ozair and Yoshua Bengio.
\newblock Deep directed generative autoencoders.
\newblock \emph{arXiv preprint arXiv:1410.0630}, 2014.

\bibitem[Richter et~al.(2016)Richter, Vineet, Roth, and
  Koltun]{Richter_2016_ECCV}
Stephan~R Richter, Vibhav Vineet, Stefan Roth, and Vladlen Koltun.
\newblock Playing for data: Ground truth from computer games.
\newblock In \emph{European Conference on Computer Vision}, pp.\  102--118.
  Springer, 2016.

\bibitem[Rodr{\'\i}guez et~al.(2005)Rodr{\'\i}guez, Alonso, and
  Maestro]{rodriguez2005support}
Juan~Jos{\'e} Rodr{\'\i}guez, Carlos~J Alonso, and Jos{\'e}~A Maestro.
\newblock Support vector machines of interval-based features for time series
  classification.
\newblock \emph{Knowledge-Based Systems}, 18\penalty0 (4):\penalty0 171--178,
  2005.

\bibitem[Schl{\"u}ter \& Grill(2015)Schl{\"u}ter and
  Grill]{schluter2015exploring}
Jan Schl{\"u}ter and Thomas Grill.
\newblock Exploring data augmentation for improved singing voice detection with
  neural networks.
\newblock In \emph{International Society for Music Information Retrieval
  Conference (ISMIR)}, 2015.

\bibitem[Shotton et~al.(2013)Shotton, Sharp, Kipman, Fitzgibbon, Finocchio,
  Blake, Cook, and Moore]{shotton2013real}
Jamie Shotton, Toby Sharp, Alex Kipman, Andrew Fitzgibbon, Mark Finocchio,
  Andrew Blake, Mat Cook, and Richard Moore.
\newblock Real-time human pose recognition in parts from single depth images.
\newblock \emph{Communications of the ACM}, 56\penalty0 (1):\penalty0 116--124,
  2013.

\bibitem[Srivastava et~al.(2015)Srivastava, Mansimov, and
  Salakhutdinov]{srivastava2015unsupervised}
Nitish Srivastava, Elman Mansimov, and Ruslan Salakhutdinov.
\newblock Unsupervised learning of video representations using lstms.
\newblock In \emph{Proceedings of the 32nd International Conference on Machine
  Learning (ICML-15)}, pp.\  843--852, 2015.

\bibitem[Sutskever et~al.(2014)Sutskever, Vinyals, and
  Le]{sutskever2014sequence}
Ilya Sutskever, Oriol Vinyals, and Quoc~V Le.
\newblock Sequence to sequence learning with neural networks.
\newblock In \emph{Advances in neural information processing systems}, pp.\
  3104--3112, 2014.

\bibitem[Venugopalan et~al.(2015)Venugopalan, Rohrbach, Donahue, Mooney,
  Darrell, and Saenko]{venugopalan2015sequence}
Subhashini Venugopalan, Marcus Rohrbach, Jeffrey Donahue, Raymond Mooney,
  Trevor Darrell, and Kate Saenko.
\newblock Sequence to sequence-video to text.
\newblock In \emph{Proceedings of the IEEE International Conference on Computer
  Vision}, pp.\  4534--4542, 2015.

\bibitem[Vinyals \& Le(2015)Vinyals and Le]{vinyals2015neural}
Oriol Vinyals and Quoc Le.
\newblock A neural conversational model.
\newblock In \emph{International Conference on Machine Learning: Deep Learning
  Workshop}, 2015.

\bibitem[Vinyals et~al.(2015{\natexlab{a}})Vinyals, Kaiser, Koo, Petrov,
  Sutskever, and Hinton]{vinyals2015grammar}
Oriol Vinyals, {\L}ukasz Kaiser, Terry Koo, Slav Petrov, Ilya Sutskever, and
  Geoffrey Hinton.
\newblock Grammar as a foreign language.
\newblock In \emph{Advances in Neural Information Processing Systems}, pp.\
  2773--2781, 2015{\natexlab{a}}.

\bibitem[Vinyals et~al.(2015{\natexlab{b}})Vinyals, Toshev, Bengio, and
  Erhan]{vinyals2015show}
Oriol Vinyals, Alexander Toshev, Samy Bengio, and Dumitru Erhan.
\newblock Show and tell: A neural image caption generator.
\newblock In \emph{Proceedings of the IEEE Conference on Computer Vision and
  Pattern Recognition}, pp.\  3156--3164, 2015{\natexlab{b}}.

\bibitem[Wan et~al.(2016)Wan, Tang, Zhang, Li, Wu, and Hoi]{wan2015hdidx}
Ji~Wan, Sheng Tang, Yongdong Zhang, Jintao Li, Pengcheng Wu, and Steven~CH Hoi.
\newblock {HD}idx: High-dimensional indexing for efficient approximate nearest
  neighbor search.
\newblock \emph{Neurocomputing}, 2016.

\bibitem[Zagoruyko \& Komodakis(2016)Zagoruyko and
  Komodakis]{zagoruyko2016wide}
Sergey Zagoruyko and Nikos Komodakis.
\newblock Wide residual networks.
\newblock In \emph{British Machine Vision Conference (BMVC) 2016}, 2016.

\end{thebibliography}
