Neural Architecture Search as Program Transformation Exploration
Jack Turner
Elliot J. Crowley
Michael F.P. O'Boyle
Improving the performance of deep neural networks (DNNs) is important to both the compiler and neural architecture search (NAS) communities. Compilers apply program transformations in order to exploit hardware parallelism and memory hierarchy. However, legality concerns mean they fail to exploit the natural robustness of neural networks. In contrast, NAS techniques mutate networks by operations such as the grouping or bottlenecking of convolutions, exploiting the resilience of DNNs. In this work, we express such neural architecture operations as program transformations whose legality depends on a notion of representational capacity. This allows them to be combined with existing transformations into a unified optimization framework. This unification allows us to express existing NAS operations as combinations of simpler transformations. Crucially, it allows us to generate and explore new tensor convolutions. We prototyped the combined framework in TVM and were able to find optimizations across different DNNs, that significantly reduce inference time - over 3$\times$ in the majority of cases. Furthermore, our scheme dramatically reduces NAS search time. Code is available at~\href{https://github.com/jack-willturner/nas-as-program-transformation-exploration{this https url.
00_introduction.tex
\section{Introduction}
\label{sec:introduction}

Deep neural networks (DNNs)~\cite{schmidhuber2015deep,lecun2015deep} are everywhere,
and there is  a growing need to implement them efficiently
~\cite{huang2019gpipe, chen2012benchnn, chowdhery2019visual}.
This has led to an explosion in research from application
~\cite{hinton2015distilling} to hardware~\cite{Chen:2014:DSH:2541940.2541967}.

Currently, there are two distinct communities optimizing  DNNs  for commodity devices.
In one, neural architecture  search (NAS) researchers explore different network models
trading off size and accuracy. In the other, compiler developers take the resulting networks
and explore optimzations to deliver hardware performance.  We argue that this  two-stage
deployment process is artificial and can be unified.  This paper shows that program
and neural architecture transformations can  be {\em interleaved}  delivering
significant performance improvement and greater expressivity. Before we describe our approach,
let us briefly look at the two existing communities.


\subsection{Program Transformations}
Within compiler research,
there has been much focus on restructuring
underlying tensor computations %to best utilize hardware
~\cite{Pu:2017:PHS:3132652.3107953, yang2016systematic, cyphers2018intel}.
This involves significant loop nest restructuring~\cite{abadi2016tensorflow,chen2018tvm}
exploiting characteristics of the  target device
e.g.\ vectorization
in SIMD units or memory coalescing in GPUs \cite{Elango:2018:DDL:3211346.3211354}.


There are currently many methods~\cite{ragan2013halide,steuwer2017lift,chen2018tvm,tillet2019triton}
in use.
The polyhedral model
~\cite{vasilache2018tensor,zerrell2019stripe}, in particular, is a natural fit for many standard neural network operations.
A key aspect of a good program  transformation
representation is that sequences of transformations can be easily composed and checked for legality. This allows automatic exploration of a large space of options which can then be evaluated, and the best selected~\cite{steuwer2017lift}.

\subsection{Neural Architecture Search}
The neural architecture search (NAS) community is also concerned with
accuracy, space, and
performance~\cite{elsken2019neural,wistuba2019survey}.
While there have been advances in NAS for language models~\cite{so2019evolved}, large scale studies of NAS tend to focus primarily on convolutional architectures~\cite{Dong2020NAS-Bench-201:,zela2020bench,ying2019bench}.
At the neural architecture level, many techniques have been proposed
for the automatic generation of networks under strict
budgets \cite{he2018amc,yang2018netadapt,wu2019fbnet,tan2019mnasnet}.
These methods focus on overall network
structure, selectively replacing components such as convolutional layers with
computationally cheaper methods to balance the trade-off between accuracy and inference time.

One key strength of NAS is that it leverages the
robustness of neural networks to deformation, reshaping and
transforming them while incurring minimal damage to their
ability to learn.
This relies on networks maintaining their ability to extract feature representations for a given type of input, which we refer to as~\textit{representational capacity}.
If an approximation or compression to a network does not inhibit its ability to learn from data, then it has not damaged the representational capacity of the network.

The ability of neural networks to weather compression without losing representational capacity is well-documented~\cite{denil2013predicting,lecun1989optimal,han2016deep}.
The legality of such neural architecture transformations, however, is not guaranteed and must be evaluated separately through either a training process or a small proxy task.
Furthermore, while powerful,
few NAS techniques explicitly take into account actual hardware behavior, and those
that do face one of three problems.
The first is that they only offer a
black box solution with highly complex methods for predicating the
search process on specific budgets (usually involving reinforcement learning~\cite{he2018amc,tan2019mnasnet}).
The second is that they do so while leaving the compilation
pipeline fixed, often dismissing powerful candidate architectures
because of an inappropriate, fixed choice of
program transformations. The final problem is that they are limited to selecting from a pre-designed list of convolutional alternatives; they cannot synthesize their own.

\subsection{Our Approach}

We have two distinct communities who have the same goal but
are siloed. NAS researchers assume the compiler is a black box bundled
with the hardware, while compiler writers assume that the network
architecture is set in stone. NAS designers can discover good
networks but are limited to pre-defined options; compiler writers can
efficiently exploit hardware structure but miss larger scale
optimization opportunities.

What we want is the best of both worlds. We wish to combine  neural architecture and compiler optimization in a unified framework. In this paper, we recast neural architecture search as program transformation exploration. By including transformations such as grouping and bottlenecking into the compiler optimization space, we leverage both the extensive history of program transformation research, and also discover new forms of neural architecture reduction that would not have been available to us otherwise.

Program transformations are necessarily restricted
as they must be safe. Our solution is to unlock the space of~\textit{neural  transformations} by introducing a new safety metric based on Fisher Potential
~\cite{Turner2020BlockSwap:}. It is a compile-time, cheap-to-compute metric that can reject damaging network changes, {\em eliminating the need to train while searching}.
We, therefore,  judge a transformation to be legal, not by data dependence preservation, but by the preservation of~\textit{representational capacity} (see section \ref{sec:fish}). This unification allows the exploration of a space that leads to more efficient implementations. It also shows that neural architecture options that previously required the engineering efforts of experts to develop, such as spatial bottlenecking, can be expressed as compositions of more fundamental transformations and discovered automatically (see section \ref{sec:spatial}).

Our contributions are as follows:
\begin{enumerate}
    \item We reformulate popular Neural Architecture Search techniques as~\textit{program transformations}.
    \item We use Fisher Potential to  provide transformation safety guarantees without the need to train.
    \item We unify the transformation and architecture search spaces, discovering new types of convolution.
    \item We evaluate 3 networks using these operations, ResNet, ResNext and DenseNet, on 4 platforms and
     demonstrate, in most cases, more than $3\times$ inference speedup over a TVM baseline.
\end{enumerate}

01_motivation.tex
\begin{figure*}[t]
    \centering
    \includegraphics[width=\linewidth]{figures/conv_fig-eps-converted-to.pdf}
    \caption{An illustration of the relation between neural architecture components, code and transformations. The first row shows a basic convolution. The second shows a tensor convolution. In the third row, we see a program transformation: loop interchange. The fourth row shows a neural architecture transformation: bottlenecking. Here, the number of weights is divided by $B$. The accuracy of the network after training does not change, but the number of operations being performed is drastically reduced.}
    \label{fig:conv}
\end{figure*}

\section{Overview}

\label{sec:overview}
Here we develop a simple example to illustrate the direct connection between models and code, and hence, neural architecture search (NAS) and program transformation. This is followed by an outline of our approach.

\subsection{Models and Code}
The fundamental building block of a DNN is the tensor convolution, a generalization of a basic convolution.

\paragraph{Basic Convolution}
Consider the first row in Figure~\ref{fig:conv}. This shows a basic convolution where each element in the 2D output matrix is the weighted sum of the corresponding and neighboring elements of the 2D input matrix. The weights are stored in a small 2D filter or weight matrix W whose size corresponds to the size of the neighborhood.  This is shown both diagrammatically on the left and as code on the right.

\paragraph{Tensor Convolution}
A tensor convolution is a generalization of the basic convolution as shown in  row 2  of Figure~\ref{fig:conv}. The input is now a 3D tensor with a new dimension, $C_i$, referred to as the input channels. Similarly, the output is expanded by the number of output channels ($C_o$). The weight matrix is now a 4D tensor of $C_o \times C_i$ channels with two spatial dimensions (height, width).
Two new  loops scan the channels, the outermost, $C_i$ shown by an arrow,  while each of the tensors arrays have appropriate extra dimensions.

\subsection{Models and Code Transformations}
In the compiler setting, transformations change structure while maintaining meaning.

\paragraph{Code Transformation}
Loop interchange changes the order of computation and memory access
without affecting the values  computed. In row 2 of Figure~\ref{fig:conv},
interchanging the outer loop iterators
gives the code in row 3 with the  $C_o$ and $C_i$ loops interchanged.
We can represent this using polyhedral notation $[C_i, C_o] \mapsto [C_o, C_i]$
or TVM-like syntax:
{\tt .interchange(CI,CO)}.
This is shown diagrammatically with the arrow denoting the new outermost, $C_o$  loop order. From a neural architecture perspective nothing has changed; the number of computations and memory accesses remains the same, as do the values of the output matrix.  From a code perspective, the memory access behavior is significantly altered, affecting execution time.

\paragraph{Model Transformation}
A popular choice in NAS for reducing convolutional complexity is bottlenecking~\cite{tan2019mnasnet}. For a convolution with $C_o$ filters, we  choose a bottlenecking factor $B$. This reduces the size of the weight and output tensors
 and reduces the range  of the~\textit{outermost loop iterator} by a factor $B$. Again this can be represented as   $[C_o] \mapsto [C'_o<C_o/B]$ or {\tt .bottleneck(B)}. This is shown in  row 4 of  Figure~\ref{fig:conv}. From a NAS point of view, this is a standard operation reducing complexity with a minimal effect on representational capacity.  From a program transformation point of view, this is illegal as the computed values are changed.


\subsection{Combined Space}
If we consider swapping a full convolution for a reduced convolution as a  program transformation,
we can combine them. For example, we could apply loop interchange to the code in Figure \ref{fig:conv} row 4, ($[C'_o<C_o/B, C_i] \mapsto [C_i, C'_o<C_o/B]$). With $C_i$  now the outermost iterator, we can reapply bottlenecking, giving input channel bottlenecking. Such an optimization is both semantically invalid, and  also unavailable in existing neural architecture search spaces. However, given the robustness of neural networks to noise, in some specific cases it may be just as representationally preserving as output channel bottlenecking. It is only through the lens of program transformations over loop nests that we are able to automatically access such operators.

02_background.tex
\section{Neural Architecture Search (NAS)}
\label{sec:nas_background}
In this section we briefly describe NAS, which seeks to automate the design of neural networks for given tasks and budgets. For more details, please see \cite{wistuba2019survey}.

Starting  from an overall  network skeleton, NAS  attempts to design one or more cells that slot into different locations within the skeleton. Cells are described as a DAG with nodes as intermediate feature maps (or tensors)  and edges representing possible operations (for example, convolution or tensor product). The task is then to find the best DAG (or cell) that can be slotted into the skeleton and trained on a given dataset.  Recent work takes cells as predefined options and attempts to select where best to place each of the cells in the skeleton~\cite{wu2019fbnet, Turner2020BlockSwap:}.
to match specific constraints.

\subsection{Neural Architectures Components}

The vast majority of neural architectures consist of sequences of inter-connected components. The convolution operation dominates computational complexity and is the primary component
of interest in NAS~\cite{ying2019bench,Dong2020NAS-Bench-201:}.
We  now introduce the convolution operation, and the variants that are considered as possible substitutions in our NAS baseline.

\paragraph{Standard Convolution}
Figure~\ref{fig:conv} row 2 shows the standard convolution operation, in which an input volume  of size $H_{i} \times W_{i} \times C_{i}$ is convolved with a set of $C_{o}$ filters of size $K_h \times K_w \times C_{i}$, each producing an individual feature map in the output.
The convolution operation is
\begin{equation}
\begin{split}
    \forall c_o, w_o, h_o \hspace{2mm} O(c_o,w_o,h_o) = \hspace{2cm} \\
    \sum_{c_i}^{C_i} \sum_{k_h}^{K_h} \sum_{k_w}^{K_w} I(c_i,w_o+k_w,h_o+k_h) * K(c_o,c_i,k_w,k_h).
\end{split}
\end{equation}
\paragraph{Bottlenecked Convolution} A popular choice for reducing convolutional complexity is bottlenecking~\cite{he2016deep}
as shown in Figure \ref{fig:conv} row 4. A bottlenecking factor $B$ is selected, reducing the number of weights/filters to $C_{o}/B$. ResNets~\cite{he2016deep} frequently feature bottleneck blocks that consist of trios of convolutions, one to bring the number of channels down, another for processing, and a final one to bring the number of channels back up. A similar pattern is relied on in several NAS techniques~\cite{tan2019mnasnet, tan2019efficientnet}.
The bottlenecked convolution operation is
\begin{equation}
\begin{split}
    \forall c'_o < \frac{C_o}{B}, w_o, h_o \hspace{2mm} O(c'_o,w_o,h_o) = \hspace{2cm} \\
    \sum_{c_i}^{C_i} \sum_{k_h}^{K_h} \sum_{k_w}^{K_w} I(c_i,w_o+k_w,h_o+k_h) * K(c'_o,c_i,k_w,k_h).
\end{split}
\end{equation}

\paragraph{Grouped Convolution}
Here, the $C_{i}$ channel input is split along the channel dimension into $G$ groups, each of which has $C_{i}/G$ channels. Each group is independently convolved with its input split, producing $C_{o}/G$ channels which are concatenated along the channel dimension.
Let $O=[O'_1;\ldots, O'_G]$ i.e. each slice  $O'_g, g \in 1,\ldots G$
is concatanated to form $O$.  $I'_g$ is similarly defined.
Group convolution is then as follows:
\begin{equation}
\begin{split}
    \forall c'_o, w_o, h_o \hspace{2mm} O(c'_o,w_o,h_o) = \hspace{2cm} \\
    \sum_{c'_i}^{C'_i} \sum_{k_h}^{K_h} \sum_{k_w}^{K_w} I(c'_i,w_o+k_w,h_o+k_h) * K(c'_o,c'_i,k_w,k_h).
\end{split}
\end{equation}

This reduces the number of basic convolutions from $C_o \times C_i$ to $G \times C_o /G \times C_i /G  = (C_o \times C_i)/G$ reducing the number of operations used by a factor of $G$. Note that we can think of standard convolution as grouped convolution with $G=1$.

\begin{figure}[t]
  \includegraphics[width=\linewidth]{figures/NAS_DAG-eps-converted-to.pdf}
  \caption{An illustration of a Neural Architecture Search design space (as used in~\cite{ Dong2020NAS-Bench-201:}). Every cell has exactly four nodes (A,B,C,D) representing intermediate feature map states. Edges represent operations that transform intermediate states from source to target node, taken from a list of options specified by the designer.
}
  \label{fig:nas-dag}
\end{figure}

\paragraph{Depthwise Convolution}
Notice that if the number of groups is equal to the number of input channels and the number of output channels, $G=C_{i}=C_{o}$, then there is a single, 2D convolutional filter for each input channel $C_i$.  This is known as depthwise convolution and is a special case of group convolution.

\subsection{NAS Search Space Example}
Typically, the NAS heuristic finds the best cell to fill in the skeleton by choosing appropriate operations. Figure~\ref{fig:nas-dag} gives an example cell from~\cite{Dong2020NAS-Bench-201:}. The skeleton is ResNet-like with 5 cells in series. Each cell contains 4 inter-connected nodes (A,B,C,D). Between nodes, downsampling takes place where the spatial dimensions are halved in size while  doubling the channel depth. This restricts the types of operations available on each edge.
This gives a total of 15625 possible %DAGs or
cells, which captures most of the available options within cell-based NAS techniques, each of which the authors train exhaustively on various datasets.

Rather than designing cells by choosing which of a predetermined list of options is best, we instead wish to choose a sequence of transformations from an original model which will allow us to step through the cell design space. We describe the program transformations we use in the next section.

\section{Program Transformations}
\label{sec:PT_background}
Due to the restricted, static,  convex and  affine nature of tensor convolutions,  it is natural to  describe program transformations of them in the  well-studied polyhedral model.
In this section we give a brief introduction to the model; for more a detailed description, we refer the reader to \cite{verdoolaege2014schedule,vasilache2018tensor}.
The polyhedral model of a program consists of three main components:

The~\textbf{domain} is a collection of the possible statement instances that occur within the iteration space  of a set of loop bounds.  We can represent each statement instance with a multidimensional co-ordinate corresponding to the iterator values of the loops that surround it.
As the constraints on the loops are affine, the  bounded domain forms a convex polyhedron.

A~\textbf{set of accesses} are affine mappings  of the iteration space to memory. Two statement instances have a dependence ordering between them if they have accesses to the same memory location and at least one of them is a write.

A~\textbf{schedule} assigns a timestamp to each statement instance, dictating the order in which they are executed. Different schedules for the same program represent possible program transformations. The transformed schedules are determined to be semantically preserving if dependence ordering is preserved.

\begin{algorithm}[t]
\begin{minted}[linenos,style=xcode]{C++}
for (co=0; co<Co; co++)
  for (oh=0; oh<OH; oh++)
    for (ow=0; w<OW; ow++)
S1    O[c_o][h][w] = 0.;
      for (ci=0; ci<Ci; ci++)
S2      O[co][oh][ow] +=
          W[co][1][1] *
          I[ci][oh][ow];
\end{minted}
\caption{Naive implementation of
$1 \times 1$ tensor convolution.
}
\label{alg:naive_res}
\end{algorithm}

To illustrate this, consider the implementation of the $1\times1$ convolution at the start of a residual block listed in Algorithm~\ref{alg:naive_res}.
We can describe the domain as follows:
\[
  \begin{array}{l}
 \begin{array}{l@{~}c@{~}l}
%                           \{
                           \mathtt{S1}(c_o,h,w)       & \mid & 0 \leq c_o < C_o \wedge 0 \leq h < H \wedge  \\ & & 0 \leq w < W
%                           \}
                           \\
%                           \{
                           \mathtt{S2}(c_o,h,w,c_i)   & \mid & 0 \leq c_o < C_o \wedge 0 \leq h < H \wedge  \\ & &  0 \leq w < W \wedge 0 \leq c_i < C_i
%                           \}
                           \\
 \end{array}
 \\
  \end{array}
 \]
We can also describe the schedule as follows:
\[
  \begin{array}{l}
    \quad\quad\quad
    T_{\mathtt{S1}}(c_o,h,w) = (c_o,h,w)
    \\
    \quad\quad\quad
    T_{\mathtt{S2}}(c_o,h,w,c_i) = (c_o,h,w,c_i)
    \\
  \end{array}
 \]
We now briefly discuss the transformations we consider.

\paragraph{\textbf{Loop Interchange}} Interchanging two nested loops involves applying a permutation to  the schedule of the enclosed statements. For example, in the algorithm shown in \ref{alg:naive_res} and a statement $S1$ we can express this as simply as:
\[
    T_{\mathtt{S1}}(c_o,h,w) = (c_o,w,h) %\}
\]

\paragraph{\textbf{Strip-Mining}} This  is performed by mapping an  iterator into two new iterators whose combined range is  that of the original. A constant  strip-mining factor is selected which forms the range of the new inner loop. The  outer loop range is that of the original divided by the strip-mine factor. For example, to strip mine the inner $c_i$ loop of Algorithm~\ref{alg:naive_res}  we have
\[
    T_{\mathtt{S2}}(c_o,h,w,c_i) = (c_o,w,h,c_i/32,c_i \Mod{32})
\]
\paragraph{\textbf{Tiling}} Tiling is a combined transformation
consisting of strip-mining followed by interchange.  For example, to tile  $c_i$ loop of \ref{alg:naive_res}  we have
\[
    T_{\mathtt{S2}}(c_o,h,w,c_i) = (c_i/32,c_o,w,h,c_i \Mod{32})
\]
There are many more transformations that can be expressed in the polyhedral model. For further detail, we point the reader to the excellent polyhedral literature~\cite{grosser2011polly,vasilache2006polyhedral}.
\subsection{Legality}
Legality in the polyhedral framework is determined by preservation of data dependences. If there is a data dependence between two dynamic statement instances under the original program  schedule, the relative ordering  between these statements must be preserved under the new transformed schedule.

In the case of Algorithm~\ref{alg:naive_res}, we can see that $S1$ is the definition, or source, of a dependence that must occur before its usage (in $S2$). For an instance $i$ of statement $S1$ and $j$ of statement $S2$, assuming constant data dependences, we can represent dependences in matrix form $D$. Elements of $D$ are non-negative integers such that an element $d_{S1, S2}$ indicates that $i$ must execute before $j$ (formally, $i \prec j$, where $\prec$ denotes lexicographical ordering).

Given a linear transformed schedule  of the iteration space $I$  in matrix form $T$ and a dependence matrix $D$, then a transformation is legal iff:
\[
\forall  i,j,S1,S2,D  \quad i \rightarrow j \in d_{S1,S2} \rightarrow T(i) \preceq T(j)
\]
which is to say that the lexicographical ordering is preserved.

03_method.tex
\section{Unified Space}
\label{sec:unified}

Our approach to joining  NAS and compiler optimizations is to describe
convolutional alternatives as polyhedral program transformations. Due to the
affine nature of convolutions, this is relatively  straightforward, except for
checking transformation legality.

We adapt a newly developed legality check based on Fisher
information~\cite{Turner2020BlockSwap:}, called Fisher Potential, that allows us
to check the legality of transformations without having to retrain the network
each time. This leads to a dramatic reduction of transformation search time as
shown in Section~\ref{sec:results}.

\subsection{Extending the Polyhedral Model}
\label{section:extending}

\paragraph{\textbf{Bottlenecking}} The bottleneck transformation is a reduction
in the outer iterator in the domain node by parameterized constant factor $B$.
Let the vector $\underline{J}$ denote the iterators spanning the domain of the
tensor loop nest, $\underline{J} = [c_o,c_i,h,w,k_h,k_w]$ where $c_o$ is the
outermost iterator and $\underline{J'} = [c_i,h,w,k_h,k_w]$ be the enclosed
inner iterators, then bottlenecking can be expressed as
\[
    T_{\mathtt{S}}(c_o,J') = (c'_o,J') \mid c'_o <C_o/B
\]
The value $B$ is constrained such that $C_o \Mod{B} \equiv 0$. This gives a factor
$B$ reduction in computation. Figure 1 row 4 shows an example of bottlenecking.

\paragraph{\textbf{Grouping}}

\begin{algorithm}[t]
\begin{minted}[linenos,style=xcode]{C++}
for (g=0; g<G; g++)
  for (co=Co/G*g; co<Co/G*(g+1); co++)
    for (ci=Ci/G*g; ci<Ci/G*(g+1); ci++)
      for (oh=0; oh<OH; oh++)
        for (ow=0; w<OW; ow++)
          for (kh=0; kh<KH; kh++)
            for (kw=0; kw<KW; kw++)
              O[co][oh][ow] +=
                        W[co][ci][kh][kw] *
                        I[co][oh+kh][ow+kw];
\end{minted}
\caption{Grouping transformation of tensor convolution.}
\label{alg:group}
\end{algorithm}

Grouping can be thought of as tiling the two outer iterators by a common factor
and then discarding one of the iterators. The original iterator domains must
both be divisible by the grouping factor $G$. Let $J' = [c_i,J'']$, such that
$J= [c_o,c_i, J'']$ then we can define grouping as
\[
    T_{\mathtt{S}}(c_o,c_i,J'') = (g,c_o/G,c_i/G,J')
\]
to give the algorithm in Figure \ref{alg:group}. Note as $C_o,C_i$ and $G$ are
compile-time known constants each of the loop bounds is affine. Examining the
code we see that each slice $g$ of the output array refers only to the
corresponding slice of the weight and input array.


\paragraph{\textbf{Depthwise}}

\begin{algorithm}[t]
\begin{minted}[linenos,style=xcode]{C++}
for (g=0; g<Co; g++)
  for (oh=0; oh<OH; oh++)
    for (ow=0; w<OW; ow++)
      for (kh=0; kh<KH; kh++)
        for (kw=0; kw<KW; kw++)
          O[g][oh][ow] +=
                    W[g][kh][kw] *
                    I[g][oh+kh][ow+kw];
\end{minted}
\caption{Depthwise transformation of tensor convolution.}
\label{alg:depthwise}
\end{algorithm}
Depthwise convolutions can be considered as a special case of group convolutions
where the group size equals the number of output channels, $C_o, G=C_o$. For
this transformation to be possible, the number of input and output channels must
be equal, $C_o=C_i$ . This means the two inner loops will have strip counts of 1
as $C_o/G= C_i/G = 1$ and
\[
    T_{\mathtt{S}}(c_o,c_i,J'') = (g,1,1,J')
\]
which can be trivially simplified to
\[
    T_{\mathtt{S}}(c_o,c_i,J'') = (g,J')
\]
\subsection{Fisher Potential as a Legality Check}
\label{sec:fish}

The transformations described above fail to preserve traditional program semantics.
We instead state that a schedule transformation for a neural network with
respect to a task is legal if the final classification accuracy on the held out
data is the same, or similar to within a small~$\delta$. Training all possible
transformed networks to determine accuracy, however, would be prohibitively
expensive.

To address this, we employ~\textit{Fisher Potential}~\cite{Turner2020BlockSwap:}
as a pre-training legality check. It is a cheap-to-compute measure that is
effective at rejecting any architectures whose final test accuracy is
significantly below the original starting point, {\em without needing to perform
training}. Informally, Fisher Potential is, locally, the total information that
each loop nest (layer) contains about class labels under a simplifying
assumption of conditional independence. We note that the specific measure
could easily be swapped out for another, such as~\cite{mellor2020neural},
or any of the measures developed in~\cite{abdelfattah2021zero}, and
that this is an area of active development.

Fisher Potential refers to the use of aggregated Fisher Information at
initialization to estimate the effectiveness of network architectures before
training. Where~\cite{lecun1989optimal} used the diagonal of the Hessian in
order to compress networks,~\cite{molchanov2017pruning,theis2018faster} showed
that this computation could be approximated via the Fisher Information Matrix.
This approximation was empirically shown to be effective at estimating the
importance of individual neurons in a neural
network~\cite{molchanov2019importance}, and to correlate to final
accuracy~\cite{golatkar2019time}. In SNIP~\cite{lee2019snip}, the authors used
a measure heavily related to Fisher Information~\textit{at initialization} in order to prune connections
in the network prior to the training process. This was adapted
in~\cite{Turner2020BlockSwap:} to perform architecture search, by summing the
Fisher Information over neurons to get layer-wise importance estimates. It is
this form of the measure that we use in this paper.

\paragraph{Formal Definition}
For a single channel, $c$, of a convolutional filter in a network, consider that
for some input minibatch of $N$ examples, its outputs are an $N \times W \times
H$ tensor where $W$ and $H$ are the channel's spatial width and height. We refer
to this tensor as the activation $A$ of  this channel and denote the  entry
corresponding to example $n$ in the mini-batch at location $(i,j)$ as $A_{nij}$.
As the network has a loss function $\mathcal{L}$, then we can get the gradient
of the loss with respect to this activation channel $\frac{\partial
\mathcal{L}}{\partial {A}}$. Let us denote this gradient as $g$ and index it as
$g_{nij}$. The channel $c$ error,  $\Delta_{c}$ can then be computed by
\begin{equation}
 \Delta_{c} =\frac{1}{2N} \sum_{n}^{N}\left(- \sum_{i}^{W}\sum_{j}^{H}A_{nij} g_{nij}\right)^2. \label{eqn:fisher}
\end{equation}
This gives us a filter-wise score for a particular channel. In order to gauge
the sensitivity of the full convolution, we sum $\Delta_{c}$ over each channel
(as in~\cite{Turner2020BlockSwap:}):
\begin{equation}
    \Delta{l} = \sum_{c_o}^{C_o} \Delta_{c_o}
\end{equation}
This score is summed for each of the convolutional blocks in the network. For an
original network and a proposed alternative architecture, we reject the proposal
if its score is below that of the original.

To summarise, the Fisher Potential of a proposed network architecture is the sum
of the Fisher Information for each layer when given a single random minibatch of
training data, as performed in~\cite{Turner2020BlockSwap:}.

\paragraph{Example}
Let us consider, candidate networks designed from
NAS-Bench-201~\cite{Dong2020NAS-Bench-201:} as shown in Figure \ref{fig:fishpo}.
Each point represents a different neural architecture of which there are 15625.
The $y$-axis shows final CIFAR-10 top-1 error (lower is better), and the
$x$-axis shows the Fisher Potential assigned to the model~\textit{at
initialization} (higher is better). We can see that without requiring any
training, Fisher Potential is able to filter out poorly-performing
architectures, visible in the cluster of low scoring networks on the left with
poor final errors. Many good networks are also discarded --- this is unfortunate but
acceptable for our scenario, since the space is large and densely populated with
networks. We note again that this measure in particular could be swapped out for an
improved one in future.


\begin{figure}[t]
\includegraphics[width=\linewidth]{figures/scores.pdf}
\caption{Fisher Potential as a rejection filter for invalid architectures. Each
point is a different neural architecture from
NAS-Bench-201~\cite{Dong2020NAS-Bench-201:}. The $y$-axis shows final CIFAR-10
top-1 error (lower is better), and the $x$-axis shows the Fisher Potential
assigned to the model~\textit{at initialization} (higher is better).}
\label{fig:fishpo}
\end{figure}


\subsection{Expressive Power}
\label{sec:spatial}
Neural Architecture Search techniques rely on hand-designed exploration spaces.
themselves are engineered by hand.
For example, a recent paper found that bottlenecking could be applied in the
spatial domain~\cite{peng2018accelerating}. This can now be added to a list of
candidate operations.
However, spatial bottlenecking  is automatically captured within our framework
as the combination of existing transformations. As an example, consider the
convolution in row 2 of Figure 1. We use the shorthand notation $[i]
\xrightarrow[]{B(b)} [i(b)]$ to denote bottlenecking domain $i$ by factor $b$
and $[i,j] \xrightarrow[]{\text{int}} [j,i]$ to denote interchange. Then the
spatial bottleneck operation can be constructed as the following sequence of
transformations:

$\displaystyle
  \begin{array}{l}
   T_S: [C_o, C_i, H, W, K_h, K_w] \xrightarrow[]{\text{int.}} \\
  \quad\quad [H, W,C_o, C_i, K_h, K_w] \xrightarrow[]{\text{B(b).}} \\
  \quad\quad [H(b), W,C_o, C_i, K_h, K_w] \xrightarrow[]{\text{int.}} \\
  \quad\quad [W,H(b),C_o, C_i, K_h, K_w] \xrightarrow[]{\text{B(b).}} \\
  \quad\quad [W(b),H(b),C_o, C_i, K_h, K_w] \xrightarrow[]{\text{int.}} \\
  \quad\quad [C_o, C_i, H(b), W(b), K_h, K_w]
  \end{array}
$
\vspace{.3cm}

The expressiveness of the polyhedral model, equipped with these new transformations, means that there are novel transformations that can be derived from this unified framework.

03b_setup.tex
\section{Implementation and Setup}
\label{sec:transf_impl}
We  expressed our new transformations as polyhedral transformations and  implemented the resulting operators in TVM~\cite{chen2018tvm}. We selected TVM as its API allows composition of transformation sequences, and is a well-accepted state-of-the-art optimizing compiler for convolutional neural networks.

\paragraph{Transformation Space}

An overview of the transformations used in this paper is given in Table~\ref{tab:opt-space}.
The first four are standard program transformations available within TVM.
To these, we add our two neural architecture transformations. Finally, for GPU platforms, we enable GPU mapping transformations.

\paragraph{Baseline TVM}
TVM allows  users  to implement schedules for each new operator by hand. Due to the large
number  of operators involved,  for each device we use TVM's default schedules;
automatic schedule design~\cite{zheng2020ansor} has yet to be incorporated into TVM.
We then enable  auto-tuning of parameter values within the  schedule to find best performance.

\paragraph{Search}
Our current search process is relatively naive. We enumerate random sequences of transformations through TVM, and generate 1000 configurations of the resulting operations for each network.
We then check which candidates pass our implementation of the Fisher Potential legality test and select the best performing one.

\begin{table}[t]
\centering
\caption{A description of the autotuning primitives available to TVM for optimizing operations, including GPU mapping, standard primitives, and our two new optimizations.}
\begin{tabular}{|p{2.3cm}|p{5.4cm}|}
\hline
    \textbf{Optimization} & \textbf{Description} \\
\hline
 \multicolumn{2}{|c|}{Program Transformations}\\
\hline

\texttt{reorder}      &  Interchange nested loops \\
\texttt{tile}         &  Cache and register blocking     \\
\texttt{unroll}       &  Loop unrolling  \\
\texttt{prefetch}     &  Memory coalescing between threads \\
\texttt{split}        &  Divide iteration into multiple axes \\
\texttt{fuse}         &  Combine two axes into one \\
\hline
  \multicolumn{2}{|c|}{Neural Architecture Transformations} \\
\hline

\texttt{bottleneck} &  Reduce domain by factor $B$ \\
\texttt{group}      &  Slice and offset two loops by factor $G$ \\

\hline
 \multicolumn{2}{|c|}{Mapping to GPU} \\
\hline
\texttt{blockIdx}     & Block-wise parallelism \\
\texttt{threadIdx}    & Threads within blocks \\
\texttt{vthread}      & Striding thread access \\

\hline

\end{tabular}


\label{tab:opt-space}
\end{table}


\subsection{Experimental Setup}
\label{sec:setup}

\paragraph{Platforms}
We evaluate on  an ARM A57 mobile CPU (mCPU), and an Nvidia 128-Core Maxwell mobile GPU (mGPU) available on the Jetson Nano board. We also evaluate  an Intel Core i7 (CPU) and an Nvidia 1080Ti (GPU). This is representative of a wide range of deployment targets, from mobile to server class. We use TVM~\texttt{v1.7} compiled with CUDA 10.1 and LLVM 8.0. For training and accuracy evaluation we use PyTorch~\texttt{v1.4}.

\paragraph{Neural Models}
We evaluate our methodology on three popular networks: ResNet-34~\cite{he2016deep}, ResNeXt-29~\cite{xie2017aggregated}, and DenseNet-161~\cite{huang2017densely}. These networks were chosen to represent a range of convolutional architectures, from standard $3\times3$ convolutions in ResNet-34 to grouped convolutions in ResNeXt and a heavy reliance on $1\times1$ convolutions in DenseNet.
CIFAR-10 models are trained for 200 epochs with Stochastic Gradient Descent (SGD)~\cite{bottou2010large} and a learning rate of 0.1, decayed by a factor of 10 at epochs 60, 120, and 160. ImageNet models were trained using the default PyTorch script\footnote{https://github.com/pytorch/examples/tree/master/imagenet}, which trains for 90 epochs with SGD, starting from a learning rate of 0.1 and decaying by a factor of 10 every 30 epochs.

\paragraph{Comparison}
The models are implemented with each operation written as a TVM Tensor Expression~\cite{chen2018tvm}, which is an einsum-style syntax for expressing tensor computations.
This is lowered to TVM IR, where our transformations can be employed. This allows for a fair comparison of each approach.

For each network, we consider three approaches. First we compile the model using
TVM using its default schedule (labeled TVM). We report the best performance
achieved after auto-tuning. Next, we use BlockSwap~\cite{Turner2020BlockSwap:}
as NAS to compress the modifiable convolutions in the network, followed by compilation with
TVM (labeled NAS). Finally, we apply our unified approach as described in the
previous section (labeled Ours).

04_results.tex
\begin{figure*}[t]
\begin{tabular}{c}
  \includegraphics[width=.9\linewidth]{figures/resnet34.pdf} \\
  (a) ResNet-34 \\
  \includegraphics[width=.9\linewidth]{figures/resnet29.pdf} \\
  (b) ResNext-29-2x64d \\
  \includegraphics[width=.9\linewidth]{figures/densenet161.pdf}\\
  (c) DenseNet-161\\
\end{tabular}
\caption{End-to-end performance for several networks on different hardware devices on CIFAR-10. TVM represents the original model compiled with TVM default schedules. The NAS columns represent the BlockSwap-compressed copies of the models
which are then compiled with TVM default schedules. Ours represents our unified NAS-compilation strategy, with new operators stacked into the network blocks.}
\label{fig:results}
\end{figure*}



\begin{figure}
\includegraphics[width=.8\linewidth]{figures/op_freq-converted-to.pdf}
\caption{Frequency of operation application.}
\label{fig:opfreq}
\end{figure}


\begin{figure*}
 \includegraphics[width=0.9\linewidth]{figures/resbench_GPU.pdf}
\caption{Exploring different sequences of transformations for an individual layer of ResNet-34 on the Intel Core i7 CPU.
~\texttt{NAS} is the result of applying grouping with factor 2 first, then compiling with TVM. The other three sequences are interleaved transformations produced by our method.}
\label{fig:resbench}
\end{figure*}

\begin{figure}
  \includegraphics[width=\linewidth]{figures/fbnet_resnet.pdf}
  \caption{Intel Core i7 performance of FBNet on the three networks. FBNet is able to improve over NAS with large training cost. Our approach outperforms FBNet with no training required }
\label{fig:more_nas}
\end{figure}

\begin{figure*}
  \includegraphics[width=\linewidth]{figures/imagenet.pdf}
  \caption{Accuracy vs inference time (log scale)  of our approach (Ours) compared to TVM (Original + TVM) when applied to  different variants of ResNet and DenseNet on the ImageNet dataset. Our approach gives a significant reduction in inference time.}
  \label{fig:imagenet}
\end{figure*}


\begin{figure}
  \includegraphics[width=\linewidth]{figures/g2_g4-eps-converted-to.pdf}
  \caption{Two NAS models (the blue points labeled NAS-A and NAS-B) composed of grouped blocks (with g=2 and g=4 respectively) can be chained together by a series of parametrized transformations in our framework, yielding the points in red. Each point is the mean of three training runs, with error bars. }
  \label{fig:g2_g4}
\end{figure}


\section{Results}
\label{sec:results}

In this section we first present the performance of the three approaches on different networks and  platforms for  the CIFAR-10 dataset. This is followed by an analysis of accuracy, size, search time and the highest performing new convolutions found.   We then drill down into one network and
examine the impact of transformations layer-by-layer.
In order to compare  against alternative NAS techniques, we evaluate the performance of FBnet across the  three networks.
Next, we  evaluate the performance of our approach on the ImageNet dataset additionally examining the tradeoff in accuracy and performance across variants of the ResNet and DenseNet family of networks. Finally we show that our approach allows for fine-grained exploration of an accuracy/size tradeoff, producing a new Pareto optimal network design.

\subsection{CIFAR-10 Network Performance}

The results for each network are shown in Figure~\ref{fig:results}. The combination of TVM and NAS is a strong baseline, however, for each of the models our unified method is able to generate models with improved hardware performance. In general there is more performance to gain on the mGPU than other platforms, as relaxed memory pressure from smaller designs is of increasing importance.

\paragraph{ResNet-34}
NAS is able to find a modest improvement of 1.12$\times$ speedup over TVM on the GPU platform, increasing to 2$\times$ on i7 CPU, showing the power of compressed convolutions.
Our unified approach  is able to find further improvement giving 2$\times$ and 3$\times$ speedup respectively. The improvement is more dramatic on the smaller platforms with 5$\times$ and 10$\times$ speedups on the mCPU and mGPU.

\paragraph{ResNext-29-2x64d}
NAS is unable to find any improvement here due to the already highly compact structure of the network. There are simply no NAS options that improve performance over the TVM
baseline across all platforms.  Despite this our approach is able to find  small improvements of 1.3$\times$ and
1.1$\times$ on the CPU and GPU platforms by combining NAS and program transformations. This increases to 1.4$\times$ on the embedded CPU and 7$\times$ on the mGPU.

\paragraph{DenseNet-161}
The impact of NAS is much more varied here. While it can find 2.2$\times$ improvement on the CPU,
it has a negligible improvement over TVM on the GPU. It also struggles on the mCPU but finds 6$\times$ on the mGPU. Our approach is able to improve by over 3$\times$ for both server platforms, but only finds 1.2$\times$ on the mCPU while acheiving 10$\times$ on the mGPU.

\subsection{Analysis}

{\bf Accuracy}
For all CIFAR-10 networks pictured in Figure~\ref{fig:results},
changes in accuracy were less than  1\% in absolute difference. The ResNet-34 in
Figure~\ref{fig:resbench} had an original ImageNet Top-1 and Top-5 accuracy of
73.2\% and 91.4\% respectively. The final network compiled with our
transformations had a Top-1 and Top-5 accuracy of 73.4\% and 91.4\%; it was
slightly more accurate than the original but considerably faster.
To give a broader overview, we present the accuracy of several ImageNet models in
Figure~\ref{fig:imagenet}, which shows that for each model accuracy
degradation is small or non-existent.

{\bf Size:} One benefit of these neural transformations is their effect on model
size, both in terms of weights and runtime memory usage. We found that CIFAR-10
networks could be compressed in size by $2-3\times$. Likewise, the ImageNet
ResNet-34 in Figure~\ref{fig:resbench} was compressed from 22M parameters to 9M
without a loss in accuracy.

{\bf Search time: } Our search process involves suggesting configurations and
rejecting or accepting them based on Fisher Potential. Since Fisher Potential is
extremely cheap to compute, the search time overhead introduced by our method is
small, less than 5 minutes on a CPU. During this time we were able to explore
1000 different configurations, discarding approximately 90\% of the candidate
transformation sequences through the Fisher Potential legality check.
\subsection{Transformation Sequence Case Studies}

There were 3 particular transformation sequences that dominated the list of best performing transformations. In neural architecture terms, the resulting operations of each of these sequences of transformations is a new convolution-like operator, previously unavailable to NAS.

~\texttt{Sequence 1} applies grouping to the kernels over the spatial domain of the input.
These are then concatenated to form one output. The exact sequence is: \texttt{[split $\rightarrow$ interchange $\rightarrow$ group $\rightarrow$ interchange $\rightarrow$ fuse]}.


~\texttt{Sequence 2} is an operator in which the output channels have been unrolled by factor $16$ and then the remaining iteration domain has been grouped by factor $G=2$.
The exact sequence is: \texttt{[unroll $\rightarrow$ group $\rightarrow$ interchange]}
~\texttt{Sequence 3} is an operator that was devised by splitting up the iteration domain of the output channels, and applying different levels of channel grouping to each new domain (e.g. $G=2$ on first half, $G=4$ on the second half).
The exact sequence is:~\texttt{[split $\rightarrow$ group $\rightarrow$ interchange $\rightarrow$ group]}.

Figure \ref{fig:opfreq} shows how often each of these sequences  appear in our best performing
networks. ResNext-29 has the fewest  instances as it contains the fewest layers, while
DenseNet-161 has the most layers and hence most instances.  Each of them is
applicable across all networks and may be more widely applicable  as standard tensor convolutions for other networks.


\subsection{Exploring Layer-Wise Optimizations}

In Figure \ref{fig:resbench}, we examine the impact of our sequences layer-by-layer on one network on one platform: ResNet-34 on the i7. The exact configurations mirror the experiment in the original TVM paper~\cite{chen2018tvm}.
While we achieve a 3x improvement overall as shown in Figure \ref{fig:results}, the speedup achieved varies considerably  across layers.
In 4 of the 11 layers no performance improvement is found, as Fisher Potential marks these individual layers to be extremely sensitive to compression.
Simple grouping with a factor $G=2$ is able to give around 2$\times$ speedup across 7 of the 11 layers.~\texttt{Sequence 1} is able to give a small improvement in  most cases --- particularly layer 11 --- where spatial reduction yields new parallelization opportunities.~\texttt{Sequence 2} provides a slight improvement in many cases, and through the later layers it is able to offer larger speedups through improved data reuse. ~\texttt{Sequence 3} is the best option for most early layers but suffers compared to other sequences in the later layers.

\subsection{Alternative NAS comparison}

To provide an alternative evaluation against an existing NAS technique, we re-implement FBNet~\cite{wu2019fbnet} using the convolutional blocks available in our NAS space, and our three baseline networks as the skeletons into which the selected cells are inserted. Figure \ref{fig:more_nas}, shows the performance of resulting networks on CIFAR-10 relative to TVM, NAS and our approach on the Intel Core i7 for each network.
In each case, FBNet does provide a modest improvement over NAS.
However, it is worth noting that this involves an expensive training step at each stage of evaluation (requiring $\sim$3 GPU days per network). Our approach is able to consistently improve over FBNet, with no training required.

\subsection{ImageNet Network Performance}

We also applied our technique to the ImageNet classification dataset, to show that the method transfers beyond CIFAR-10. In Figure~\ref{fig:imagenet}, we first show the resultant networks for running our method on ResNet-18, ResNet-34, DenseNet-161, DenseNet-201, and DenseNet-169, with their inference times recorded on the Intel i7 CPU. For each network, accuracy is within 2\%, and there are significant gains in inference time.

\subsection{Interpolating Between Models}

The additional expressive power of our method allows us to explore the search space of networks in a more fine-grained manner than traditional NAS techniques. This is because NAS techniques simply choose operations from a list, whereas we generate new ones. To illustrate this consider Figure~\ref{fig:g2_g4} where we plot the accuracy and inference time of a ResNet-34
with two BlockSwap-based models (the blue points labeled NAS-A and NAS-B). We can explore alternatives by a series of parametrized transformations in our framework, yielding the points in red. Each of these points is a new possible blocktype that would not be accessible to a traditional NAS technique unless explicitly written by the human designer.
During this interpolation we are also able to find a Pareto optimal point.

05_related_work.tex
\section{Related Work}
\label{sec:related_work}
\paragraph{Compiler optimization}
There has been much interest in  autotuning DNN code
generators~\cite{jia2019taso,chen2018tvm,vasilache2018tensor,steuwer2017lift, gibson2020optimizing, mogers2020automatic}.
Polyhedral compilers are particularly well-suited
~\cite{vasilache2018tensor,zerrell2019stripe} as they have in-built abstractions
for exploiting parallelism and memory layout in a principled form. Polyhedral
compilation itself benefits from a wealth of
literature~\cite{kelly1998framework,cohen1999analyse,verdoolaege2014schedule,vasilache2006polyhedral,Girbal2006,uday08}
that describes a unified compilation scheme and  associated set of legality
checks delivering state-of-the-art performance across a wide range of standard
compiler
benchmarks~\cite{verdoolaege2010isl,grosser2011polly,verdoolaege2013polyhedral},
as well as image and neural network specific
ones~\cite{mullapudi2015polymage,vasilache2018tensor,zerrell2019stripe}.

Other popular approaches include loop synthesis~\cite{chen2018tvm} and rewrite
rules~\cite{steuwer2017lift} which optimize through parametrized schedules. In
particular, TVM~\cite{chen2018tvm} is an adaptation of
Halide~\cite{ragan2013halide} with specific extensions for deep neural network
abstractions. Triton~\cite{tillet2019triton} is a tile-based IR for
parallelizing tensor-based computation. Most of these frameworks focus on
operator-level optimization, though higher level graph transformation has also
shown promising results ~\cite{jia2019taso}.

Though the toolchains are exceptionally feature-rich, there is evidence that
some implementations have been highly-engineered for specific workloads, at the cost of general
support for newer optimizations~\cite{barham2019machine}. All current
approaches, however, are limited by their inability to exploit NAS
transformations.

Reducing the amount of computation at the cost of accuracy has been examined in
a compiler context using loop
perforation~\cite{sidiroglou2011managing,sampson2015accept,figurnov2015perforatedcnns} in approximate
computing. Within approximate computing, there is extensive work on probabilistic
program transformations~\cite{rinard2006probabilistic,misailovic2011probabilistically,zhu2012randomized}
and language support for approximation~\cite{goiri2015approxhadoop,park2015flexjava}.
Where some approximate computing methods accept small absolute differences
in the outputs of programs, our transformations may render the numerical outputs of
our programs completely different while maintaining the legality of the program.
The benefit of Fisher Potential, while it does not give any guarantees on behavior,
is that its domain specificity allows us to capture these transformations.

\paragraph{Model Optimization}
From a machine-learning perspective, deep neural networks should be highly
compressible as there is significant evidence that they are vastly
overparametrized~\cite{denil2013predicting,frankle2019lottery}. There are
several popular strategies that can be employed. One can prune a network by
removing unimportant connections between
weights~\cite{han2016deep,gale2019state,molchanov2017pruning}. However, the
sparsity introduced can lead to poor memory behavior \cite{crowley2018pruning}. An alternative is to
retain dense weight tensors by pruning channels~\cite{li2017pruning} although
the resulting irregular structures cause a significant slowdown \cite{radu2019performance}.

Network distillation~\cite{ba2014do,hinton2015distilling, turner2018distilling},  takes a trained
large network and uses its outputs to assist in the training of a smaller
network. This, however, leads to a large design space as there are many ways to
make a network smaller
~\cite{chollet2017xception,xie2017aggregated,ioannou2017deep,gao2018condensenet,howard2017mobilenets}.

~\textit{Neural architecture search} instead automates the process of finding
efficient networks. In~\cite{zoph2017neural} the authors use an RNN to generate
network descriptions and filter the options using reinforcement learning.
Several networks are generated and trained in parallel, which requires a large
quantity of possible networks to be stored at once. To reduce this overhead, an
alternative method is to design one~\textit{supernet} that contains all of the
possible subnetworks~\cite{pham2018efficient}. This allows all models to have
access to a shared set of weights, reducing the computational requirement.
Subsequent works have made extensive use of this
technique~\cite{liu2019darts,luo2018neural,chen2018searching,wu2019fbnet}.

However, there is some evidence to suggest that weight sharing schemes
aggressively hamper the ability of the NAS agent to identify good architectures,
and under such constrained spaces, random architecture search provides a
competitive baseline~\cite{li2019random,sciuto2019evaluating}. Although NAS is a
rapidly developing area, no-one to the best of our knowledge has considered
formalizing NAS as program transformation.

06_conclusion.tex
\section{Conclusions and Future Work}
\label{sec:conclusion}

This paper presents a new unified program transformation approach to optimizing convolution neural networks by expressing neural model operations as formal
program transformations.
We  develop a new legality check based on Fisher potential,
and  show that different types of convolution can be described as
transformations of more fundamental building blocks.
We implemented this approach in TVM and show that our combined approach significantly
outperforms both TVM   and state-of-the-art NAS.
We eliminate the need to train while searching, dramatically reducing search time.
Future work could consider further transformations in the neural architecture space in order for a more extensive analysis, and to expand the possibilities beyond that of just convolutional networks.

main.tex
%% This is file `sample-authordraft.tex',
%% generated with the docstrip utility.
%%
%% The original source files were:
%%
%% samples.dtx  (with options: `authordraft')
%%
%% IMPORTANT NOTICE:
%%
%% For the copyright see the source file.
%%
%% Any modified versions of this file must be renamed
%% with new filenames distinct from sample-authordraft.tex.
%%
%% For distribution of the original source see the terms
%% for copying and modification in the file samples.dtx.
%%
%% This generated file may be distributed as long as the
%% original source files, as listed above, are part of the
%% same distribution. (The sources need not necessarily be
%% in the same archive or directory.)
%%
%% The first command in your LaTeX source must be the \documentclass command.
\documentclass[sigconf,screen]{acmart}
%% NOTE that a single column version may be required for
%% submission and peer review. This can be done by changing
%% the \doucmentclass[...]{acmart} in this template to
%% \documentclass[manuscript,screen,review]{acmart}
%%
%% To ensure 100% compatibility, please check the white list of
%% approved LaTeX packages to be used with the Master Article Template at
%% https://www.acm.org/publications/taps/whitelist-of-latex-packages
%% before creating your document. The white list page provides
%% information on how to submit additional LaTeX packages for
%% review and adoption.
%% Fonts used in the template cannot be substituted; margin
%% adjustments are not allowed.
%%
%% \BibTeX command to typeset BibTeX logo in the docs
\AtBeginDocument{%
  \providecommand\BibTeX{{%
    \normalfont B\kern-0.5em{\scshape i\kern-0.25em b}\kern-0.8em\TeX}}}

%%% If you see 'ACMUNKNOWN' in the 'setcopyright' statement below,
%%% please first submit your publishing-rights agreement with ACM (follow link on submission page).
%%% Then please update our instructions page and copy-and-paste the NEW commands into your article.
%%% Please contact us in case of questions; allow up to 10 min for the system to propagate the information.
%%%
%%% The following is specific to ASPLOS '21 and the paper
%%% 'Neural Architecture Search as Program Transformation Exploration'
%%% by Jack Turner, Elliot J Crowley, and Michael F. P. O'Boyle.
%%%
\setcopyright{acmlicensed}
\acmPrice{15.00}
\acmDOI{10.1145/3445814.3446753}
\acmYear{2021}
\copyrightyear{2021}
\acmSubmissionID{asplos21main-p969-p}
\acmISBN{978-1-4503-8317-2/21/04}
\acmConference[ASPLOS '21]{Proceedings of the 26th ACM International Conference on Architectural Support for Programming Languages and Operating Systems}{April 19--23, 2021}{Virtual, USA}
\acmBooktitle{Proceedings of the 26th ACM International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS '21), April 19--23, 2021, Virtual, USA}

%%
%% Submission ID.
%% Use this when submitting an article to a sponsored event. You'll
%% receive a unique submission ID from the organizers
%% of the event, and this ID should be used as the parameter to this command.
%%\acmSubmissionID{123-A56-BU3}

%%
%% The majority of ACM publications use numbered citations and
%% references.  The command \citestyle{authoryear} switches to the
%% "author year" style.
%%
%% If you are preparing content for an event
%% sponsored by ACM SIGGRAPH, you must use the "author year" style of
%% citations and references.
%% Uncommenting
%% the next command will enable that style.
%%\citestyle{acmauthoryear}

%\usepackage{amsmath,amssymb,amsfonts}
\newcommand{\Mod}[1]{\ (\mathrm{mod}\ #1)}
\usepackage{algorithmic}
\usepackage{algorithm}
\usepackage{graphicx}
\usepackage{textcomp}

%% personal packages
\usepackage{verbatim}
\usepackage[frozencache=true]{minted}
\usepackage{shortbold}
\usepackage{caption}

%%
%% end of the preamble, start of the body of the document source.
\begin{document}

%%
%% The "title" command has an optional parameter,
%% allowing the author to define a "short title" to be used in page headers.
\title{Neural Architecture Search as Program Transformation Exploration}

%%
%% The "author" command and its associated commands are used to define
%% the authors and their affiliations.
%% Of note is the shared affiliation of the first two authors, and the
%% "authornote" and "authornotemark" commands
%% used to denote shared contribution to the research.

\author{Jack Turner}
\email{jack.turner@ed.ac.uk}
\affiliation{%
    \institution{University of Edinburgh}
    \country{United Kingdom}
}

\author{Elliot J. Crowley}
\email{elliot.j.crowley@ed.ac.uk}
\affiliation{%
    \institution{University of Edinburgh}
    \country{United Kingdom}
}

\author{Michael  F.P. O'Boyle}
\email{mob@inf.ed.ac.uk}
\affiliation{%
    \institution{University of Edinburgh}
    \country{United Kingdom}
}

%%
%% By default, the full list of authors will be used in the page
%% headers. Often, this list is too long, and will overlap
%% other information printed in the page headers. This command allows
%% the author to define a more concise list
%% of authors' names for this purpose.

%%
%% The abstract is a short summary of the work to be presented in the
%% article.
\begin{abstract}
    Improving the performance of deep neural networks (DNNs) is important to both the compiler  and neural architecture search (NAS) communities. Compilers apply program transformations in order to  exploit hardware parallelism and memory hierarchy. However, legality concerns mean they fail to exploit the natural robustness of neural networks. In contrast, NAS techniques mutate networks by operations such as the grouping or bottlenecking of convolutions, exploiting the resilience of DNNs.
    In this work, we express  such neural architecture operations as program transformations whose legality depends on a notion of representational capacity. This allows them to be combined with existing  transformations into a unified optimization framework. This unification allows us to express existing NAS operations
    as combinations of simpler transformations. Crucially, it allows us to generate and explore new tensor convolutions. We prototyped the combined framework in TVM and  were able to find optimizations across different DNNs,  that  significantly reduce  inference time - over 3$\times$ in the majority of cases.
    Furthermore, our scheme  dramatically reduces NAS search time. Code is available at~\href{https://github.com/jack-willturner/nas-as-program-transformation-exploration}{this https url}.
\end{abstract}

%%
%% The code below is generated by the tool at http://dl.acm.org/ccs.cfm.
%% Please copy and paste the code instead of the example below.
%%
\begin{CCSXML}
    <ccs2012>
       <concept>
           <concept_id>10010147.10010257</concept_id>
           <concept_desc>Computing methodologies~Machine learning</concept_desc>
           <concept_significance>500</concept_significance>
           </concept>
       <concept>
           <concept_id>10011007.10011006.10011041</concept_id>
           <concept_desc>Software and its engineering~Compilers</concept_desc>
           <concept_significance>500</concept_significance>
           </concept>
       <concept>
           <concept_id>10010147.10010257</concept_id>
           <concept_desc>Computing methodologies~Machine learning</concept_desc>
           <concept_significance>500</concept_significance>
           </concept>
     </ccs2012>
\end{CCSXML}

\ccsdesc[500]{Computing methodologies~Machine learning}
\ccsdesc[500]{Software and its engineering~Compilers}


%%
%% Keywords. The author(s) should pick words that accurately describe
%% the work being presented. Separate the keywords with commas.
\keywords{program transformations, neural networks}


%%
%% This command processes the author and affiliation and title
%% information and builds the first part of the formatted document.
\maketitle

\input{00_introduction.tex}

\input{01_motivation.tex}

\input{02_background.tex}

\input{03_method.tex}

\input{03b_setup.tex}

\input{04_results.tex}

\input{05_related_work.tex}

\input{06_conclusion.tex}


\begin{acks}
The material is partially based on research by Michael O'Boyle which was sponsored
by  Defense Advanced Research Projects Agency (DARPA) under agreement number FA8650-18-2-7864. The views and conclusions contained herein are those of the authors and do
not represent the official policies or endorsements, either expressed
or implied, of DARPA or the U.S. Government.
This work was partially supported by the Engineering and Physical Sciences Research Council (grant EP/L0150), EPSRC Centre for Doctoral Training in Pervasive Parallelism at the University of Edinburgh, School of Informatics.
\end{acks}

%%
%% The next two lines define the bibliography style to be used, and
%% the bibliography file.
\bibliographystyle{ACM-Reference-Format}
\bibliography{acmart}


\end{document}
\endinput
%%
%% End of file `sample-authordraft.tex'.

acmart.bib
@inproceedings{gibson2020optimizing,
	title        = {Optimizing Grouped Convolutions on Edge Devices},
	author       = {Gibson, Perry and Cano, Jos{\'e} and Turner, Jack and Crowley, Elliot J and O’Boyle, Michael and Storkey, Amos},
	year         = 2020,
	booktitle    = {2020 IEEE 31st International Conference on Application-specific Systems, Architectures and Processors (ASAP)},
	pages        = {189--196},
	organization = {IEEE}
}
@article{abdelfattah2021zero,
	title        = {Zero-Cost Proxies for Lightweight NAS},
	author       = {Abdelfattah, Mohamed S and Mehrotra, Abhinav and Dudziak, {\L}ukasz and Lane, Nicholas D},
	year         = 2021,
	journal      = {arXiv preprint arXiv:2101.08134}
}
@inproceedings{mogers2020automatic,
	title        = {Automatic generation of specialized direct convolutions for mobile GPUs},
	author       = {Mogers, Naums and Radu, Valentin and Li, Lu and Turner, Jack and O'Boyle, Michael and Dubach, Christophe},
	year         = 2020,
	booktitle    = {Proceedings of the 13th Annual Workshop on General Purpose Processing using Graphics Processing Unit},
	pages        = {41--50}
}
@inproceedings{radu2019performance,
	title        = {Performance aware convolutional neural network channel pruning for embedded GPUs},
	author       = {Radu, Valentin and Kaszyk, Kuba and Wen, Yuan and Turner, Jack and Cano, Jos{\'e} and Crowley, Elliot J and Franke, Bj{\"o}rn and Storkey, Amos and O'Boyle, Michael},
	year         = 2019,
	booktitle    = {2019 IEEE International Symposium on Workload Characterization (IISWC)},
	pages        = {24--34},
	organization = {IEEE}
}
@article{crowley2018pruning,
	title        = {Pruning neural networks: is it time to nip it in the bud?},
	author       = {Crowley, Elliot J and Turner, Jack and Storkey, Amos and O'Boyle, Michael},
	year         = 2018,
  journal      = {arXiv preprint arXiv:1810.04622}
}
@article{turner2018distilling,
	title        = {Distilling with performance enhanced students},
	author       = {Turner, Jack and Crowley, Elliot J and Radu, Valentin and Cano, Jos{\'e} and Storkey, Amos and O'Boyle, Michael},
	year         = 2018,
	journal      = {arXiv preprint arXiv:1810.10460}
}
@article{turner2018distilling,
	title        = {Distilling with performance enhanced students},
	author       = {Turner, Jack and Crowley, Elliot J and Radu, Valentin and Cano, Jos{\'e} and Storkey, Amos and O'Boyle, Michael},
	year         = 2018,
	journal      = {arXiv preprint arXiv:1810.10460}
}
@misc{codeRef,
	title        = {Blind review},
	author       = {Withheld}
}
@inproceedings{anonymous2021neural,
	title        = {Neural Architecture Search on ImageNet in Four {\{}GPU{\}} Hours: A Theoretically Inspired Perspective},
	author       = {Anonymous},
	year         = 2021,
	booktitle    = {Submitted to International Conference on Learning Representations},
	url          = {https://openreview.net/forum?id=Cnon5ezMHtu},
	note         = {under review}
}
@inproceedings{rinard2006probabilistic,
	title        = {Probabilistic accuracy bounds for fault-tolerant computations that discard tasks},
	author       = {Rinard, Martin},
	year         = 2006,
	booktitle    = {Proceedings of the 20th annual international conference on Supercomputing},
	pages        = {324--334}
}
@inproceedings{chaudhuri2011proving,
	title        = {Proving programs robust},
	author       = {Chaudhuri, Swarat and Gulwani, Sumit and Lublinerman, Roberto and Navidpour, Sara},
	year         = 2011,
	booktitle    = {Proceedings of the 19th ACM SIGSOFT symposium and the 13th European conference on Foundations of software engineering},
	pages        = {102--112}
}
@inproceedings{misailovic2011probabilistically,
	title        = {Probabilistically accurate program transformations},
	author       = {Misailovic, Sasa and Roy, Daniel M and Rinard, Martin C},
	year         = 2011,
	booktitle    = {International Static Analysis Symposium},
	pages        = {316--333},
	organization = {Springer}
}
@article{zhu2012randomized,
	title        = {Randomized accuracy-aware program transformations for efficient approximate computations},
	author       = {Zhu, Zeyuan Allen and Misailovic, Sasa and Kelner, Jonathan A and Rinard, Martin},
	year         = 2012,
	journal      = {ACM SIGPLAN Notices},
	publisher    = {ACM New York, NY, USA},
	volume       = 47,
	number       = 1,
	pages        = {441--454}
}
@article{carbin2012proving,
	title        = {Proving acceptability properties of relaxed nondeterministic approximate programs},
	author       = {Carbin, Michael and Kim, Deokhwan and Misailovic, Sasa and Rinard, Martin C},
	year         = 2012,
	journal      = {ACM SIGPLAN Notices},
	publisher    = {ACM New York, NY, USA},
	volume       = 47,
	number       = 6,
	pages        = {169--180}
}
@inproceedings{carbin2013verified,
	title        = {Verified integrity properties for safe approximate program transformations},
	author       = {Carbin, Michael and Kim, Deokhwan and Misailovic, Sasa and Rinard, Martin C},
	year         = 2013,
	booktitle    = {Proceedings of the ACM SIGPLAN 2013 workshop on Partial evaluation and program manipulation},
	pages        = {63--66}
}
@article{figurnov2015perforatedcnns,
	title        = {Perforatedcnns: Acceleration through elimination of redundant convolutions},
	author       = {Figurnov, Michael and Ibraimova, Aijan and Vetrov, Dmitry and Kohli, Pushmeet},
	year         = 2015,
	journal      = {arXiv preprint arXiv:1504.08362}
}
@inproceedings{ringenburg2015monitoring,
	title        = {Monitoring and debugging the quality of results in approximate programs},
	author       = {Ringenburg, Michael and Sampson, Adrian and Ackerman, Isaac and Ceze, Luis and Grossman, Dan},
	year         = 2015,
	booktitle    = {Proceedings of the Twentieth International Conference on Architectural Support for Programming Languages and Operating Systems},
	pages        = {399--411}
}
@inproceedings{goiri2015approxhadoop,
	title        = {Approxhadoop: Bringing approximations to mapreduce frameworks},
	author       = {Goiri, Inigo and Bianchini, Ricardo and Nagarakatte, Santosh and Nguyen, Thu D},
	year         = 2015,
	booktitle    = {Proceedings of the Twentieth International Conference on Architectural Support for Programming Languages and Operating Systems},
	pages        = {383--397}
}
@inproceedings{park2015flexjava,
	title        = {Flexjava: Language support for safe and modular approximate programming},
	author       = {Park, Jongse and Esmaeilzadeh, Hadi and Zhang, Xin and Naik, Mayur and Harris, William},
	year         = 2015,
	booktitle    = {Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering},
	pages        = {745--757}
}
@article{stanley2018exploiting,
	title        = {Exploiting errors for efficiency: A survey from circuits to algorithms},
	author       = {Stanley-Marbell, Phillip and Alaghi, Armin and Carbin, Michael and Darulova, Eva and Dolecek, Lara and Gerstlauer, Andreas and Gillani, Ghayoor and Jevdjic, Djordje and Moreau, Thierry and Cacciotti, Mattia and others},
	year         = 2018,
	journal      = {arXiv preprint arXiv:1809.05859}
}
@article{sampson2015accept,
	title        = {Accept: A programmer-guided compiler framework for practical approximate computing},
	author       = {Sampson, Adrian and Baixo, Andr{\'e} and Ransford, Benjamin and Moreau, Thierry and Yip, Joshua and Ceze, Luis and Oskin, Mark},
	year         = 2015,
	journal      = {University of Washington Technical Report UW-CSE-15-01},
	volume       = 1,
	number       = 2
}
@inproceedings{golatkar2019time,
	title        = {Time matters in regularizing deep networks: Weight decay and data augmentation affect early learning dynamics, matter little near convergence},
	author       = {Golatkar, Aditya Sharad and Achille, Alessandro and Soatto, Stefano},
	year         = 2019,
	booktitle    = {Advances in Neural Information Processing Systems},
	pages        = {10678--10688}
}
@article{mellor2020neural,
	title        = {Neural Architecture Search without Training},
	author       = {Mellor, Joseph and Turner, Jack and Storkey, Amos and Crowley, Elliot J},
	year         = 2020,
	journal      = {arXiv preprint arXiv:2006.04647}
}
@inproceedings{molchanov2019importance,
	title        = {Importance estimation for neural network pruning},
	author       = {Molchanov, Pavlo and Mallya, Arun and Tyree, Stephen and Frosio, Iuri and Kautz, Jan},
	year         = 2019,
	booktitle    = {Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition},
	pages        = {11264--11272}
}
@article{zheng2020ansor,
	title        = {Ansor: Generating High-Performance Tensor Programs for Deep Learning},
	author       = {Zheng, Lianmin and Jia, Chengfan and Sun, Minmin and Wu, Zhao and Yu, Cody Hao and Haj-Ali, Ameer and Wang, Yida and Yang, Jun and Zhuo, Danyang and Sen, Koushik and others},
	year         = 2020,
	journal      = {arXiv preprint arXiv:2006.06762}
}
@inproceedings{sidiroglou2011managing,
	title        = {Managing performance vs. accuracy trade-offs with loop perforation},
	author       = {Sidiroglou-Douskos, Stelios and Misailovic, Sasa and Hoffmann, Henry and Rinard, Martin},
	year         = 2011,
	booktitle    = {Proceedings of the 19th ACM SIGSOFT symposium and the 13th European conference on Foundations of software engineering},
	pages        = {124--134}
}
@inproceedings{Chen:2014:DSH:2541940.2541967,
	title        = {DianNao: A Small-footprint High-throughput Accelerator for Ubiquitous Machine-learning},
	author       = {Chen, Tianshi and Du, Zidong and Sun, Ninghui and Wang, Jia and Wu, Chengyong and Chen, Yunji and Temam, Olivier},
	year         = 2014,
	booktitle    = {Proceedings of the International Conference on Architectural Support for Programming Languages and Operating Systems}
}
@inproceedings{uday08,
	title        = {A Practical Automatic Polyhedral Parallelizer and Locality Optimizer},
	author       = {Bondhugula, Uday and Hartono, Albert and Ramanujam, J. and Sadayappan, P.},
	year         = 2008,
	booktitle    = {Proceedings of the 29th ACM SIGPLAN Conference on Programming Language Design and Implementation},
	location     = {Tucson, AZ, USA},
	publisher    = {Association for Computing Machinery},
	address      = {New York, NY, USA},
	series       = {PLDI ’08},
	pages        = {101–113},
	doi          = {10.1145/1375581.1375595},
	isbn         = 9781595938602,
	url          = {https://doi.org/10.1145/1375581.1375595},
	numpages     = 13,
	keywords     = {automatic parallelization, affine transformations, polyhedral model, locality optimization, tiling, loop transformations}
}
@inproceedings{mullapudi2015polymage,
	title        = {Polymage: Automatic optimization for image processing pipelines},
	author       = {Mullapudi, Ravi Teja and Vasista, Vinay and Bondhugula, Uday},
	year         = 2015,
	booktitle    = {ACM SIGPLAN Notices},
	volume       = 50,
	number       = 4,
	pages        = {429--443},
	organization = {ACM}
}
@misc{elsken2019neural,
	title        = {Neural Architecture Search.},
	author       = {Elsken, Thomas and Metzen, Jan Hendrik and Hutter, Frank and others},
	year         = 2019
}
@article{so2019evolved,
	title        = {The evolved transformer},
	author       = {So, David R and Liang, Chen and Le, Quoc V},
	year         = 2019,
	journal      = {arXiv preprint arXiv:1901.11117}
}
@inproceedings{cummins2017end,
	title        = {End-to-end deep learning of optimization heuristics},
	author       = {Cummins, Chris and Petoumenos, Pavlos and Wang, Zheng and Leather, Hugh},
	year         = 2017,
	booktitle    = {2017 26th International Conference on Parallel Architectures and Compilation Techniques (PACT)},
	pages        = {219--232},
	organization = {IEEE}
}
@inproceedings{DBLP:conf/ispass/ParasharRSCYMVK19,
	title        = {Timeloop: {A} Systematic Approach to {DNN} Accelerator Evaluation},
	author       = {Angshuman Parashar and Priyanka Raina and Yakun Sophia Shao and Yu{-}Hsin Chen and Victor A. Ying and Anurag Mukkara and Rangharajan Venkatesan and Brucek Khailany and Stephen W. Keckler and Joel S. Emer},
	year         = 2019,
	booktitle    = {{IEEE} International Symposium on Performance Analysis of Systems and Software, {ISPASS} 2019, Madison, WI, USA, March 24-26, 2019},
	pages        = {304--315},
	doi          = {10.1109/ISPASS.2019.00042},
	url          = {https://doi.org/10.1109/ISPASS.2019.00042},
	crossref     = {DBLP:conf/ispass/2019},
	timestamp    = {Sat, 19 Oct 2019 20:10:45 +0200},
	biburl       = {https://dblp.org/rec/bib/conf/ispass/ParasharRSCYMVK19},
	bibsource    = {dblp computer science bibliography, https://dblp.org}
}
@article{Pu:2017:PHS:3132652.3107953,
	title        = {Programming Heterogeneous Systems from an Image Processing DSL},
	author       = {Pu, Jing and Bell, Steven and Yang, Xuan and Setter, Jeff and Richardson, Stephen and Ragan-Kelley, Jonathan and Horowitz, Mark},
	year         = 2017,
	month        = aug,
	journal      = {ACM Trans. Archit. Code Optim.},
	publisher    = {ACM},
	address      = {New York, NY, USA},
	volume       = 14,
	number       = 3,
	pages        = {26:1--26:25},
	doi          = {10.1145/3107953},
	issn         = {1544-3566},
	url          = {http://doi.acm.org/10.1145/3107953},
	issue_date   = {September 2017},
	articleno    = 26,
	numpages     = 25,
	acmid        = 3107953,
	keywords     = {FPGAs, Image processing, domain specific languages, high-level synthesis}
}
@article{yang2016systematic,
	title        = {A Systematic Approach to Blocking Convolutional Neural Networks},
	author       = {Xuan Yang and Jing Pu and Blaine Burton Rister and Nikhil Bhagdikar and Stephen Richardson and Shahar Kvatinsky and Jonathan Ragan-Kelley and Ardavan Pedram and Mark Horowitz},
	year         = 2016,
	journal      = {arXiv preprint arXiv:1606.04209}
}
@article{cyphers2018intel,
	title        = {Intel nGraph: An Intermediate Representation, Compiler, and Executor for Deep Learning},
	author       = {Scott Cyphers and Arjun K. Bansal and Anahita Bhiwandiwalla and Jayaram Bobba and Matthew Brookhart and Avijit Chakraborty and Will Constable and Christian Convey and Leona Cook and Omar Kanawi and Robert Kimball and Jason Knight and Nikolay Korovaiko and Varun Kumar and Yixing Lao and Christopher R. Lishka and Jaikrishnan Menon and Jennifer Myers and Sandeep Aswath Narayana and Adam Procter and Tristan J. Webb},
	year         = 2018,
	journal      = {arXiv preprint arXiv:1801.08058}
}
,
@inproceedings{Elango:2018:DDL:3211346.3211354,
	title        = {Diesel: DSL for Linear Algebra and Neural Net Computations on GPUs},
	author       = {Elango, Venmugil and Rubin, Norm and Ravishankar, Mahesh and Sandanagobalane, Hariharan and Grover, Vinod},
	year         = 2018,
	booktitle    = {In International Workshop on Machine Learning and Programming Languages}
}
@inproceedings{Ragan-Kelley:2013:HLC:2491956.2462176,
	title        = {Halide: A Language and Compiler for Optimizing Parallelism, Locality, and Recomputation in Image Processing Pipelines},
	author       = {Ragan-Kelley, Jonathan and Barnes, Connelly and Adams, Andrew and Paris, Sylvain and Durand, Fr{\'e}do and Amarasinghe, Saman},
	year         = 2013,
	booktitle    = {Proceedings of the 34th ACM SIGPLAN Conference on Programming Language Design and Implementation},
	location     = {Seattle, Washington, USA},
	publisher    = {ACM},
	address      = {New York, NY, USA},
	series       = {PLDI '13},
	pages        = {519--530},
	doi          = {10.1145/2491956.2462176},
	isbn         = {978-1-4503-2014-6},
	url          = {http://doi.acm.org/10.1145/2491956.2462176},
	numpages     = 12,
	acmid        = 2462176,
	keywords     = {autotuning, compiler, domain specific language, gpu, image processing, locality, optimization, parallelism, redundant computation, vectorization}
}
@inproceedings{Steuwer:2017:LFD:3049832.3049841,
	title        = {Lift: A Functional Data-parallel IR for High-performance GPU Code Generation},
	author       = {Steuwer, Michel and Remmelg, Toomas and Dubach, Christophe},
	year         = 2017,
	booktitle    = {International Symposium on Code Generation and Optimization}
}
@article{vasilache2018tensor,
	title        = {Tensor Comprehensions: Framework-Agnostic High-Performance Machine Learning Abstractions},
	author       = {Nicolas Vasilache and Oleksandr Zinenko and Theodoros Theodoridis and Priya Goyal and Zachary DeVito and William S. Moses and Sven Verdoolaege and Andrew Adams and Albert Cohen},
	year         = 2018,
	journal      = {arXiv preprint arXiv:1802.04730}
}
@techreport{gilbert1955theory,
	title        = {Theory of Shuffling},
	author       = {Gilbert, Edgar},
	year         = 1955,
	address      = {Murray Hill, New Jersey, U.S.},
	institution  = {Bell Labs}
}
@inproceedings{lecun1989optimal,
	title        = {Optimal Brain Damage},
	author       = {LeCun, Yann and John S. Denker and Sara A. Solla},
	year         = 1989,
	booktitle    = {Advances in Neural Information Processing Systems},
	pages        = {}
}
@inproceedings{hassibi1993second,
	title        = {Second order derivatives for network pruning: Optimal brain surgeon},
	author       = {Hassibi, Babak and Stork, David G.},
	year         = 1993,
	booktitle    = {Advances in Neural Information Processing Systems},
	pages        = {}
}
@techreport{kelly1998framework,
	title        = {A framework for unifying reordering transformations},
	author       = {Kelly, Wayne and Pugh, William},
	year         = 1998
}
@phdthesis{cohen1999analyse,
	title        = {Analyse et transformation de programmes: du mod{\`e}le poly{\'e}drique aux langages formels},
	author       = {Cohen, Albert Henri},
	year         = 1999,
	school       = {Versailles-St Quentin en Yvelines}
}
@article{Girbal2006,
	title        = {Semi-Automatic Composition of Loop Transformations for Deep Parallelism and Memory Hierarchies},
	author       = {Girbal, Sylvain and Vasilache, Nicolas and Bastoul, C{\'e}dric and Cohen, Albert and Parello, David and Sigler, Marc and Temam, Olivier},
	year         = 2006,
	month        = jun,
	day          = {01},
	journal      = {International Journal of Parallel Programming},
	volume       = 34,
	number       = 3,
	pages        = {261--317},
	doi          = {10.1007/s10766-006-0012-3},
	issn         = {1573-7640},
	url          = {https://doi.org/10.1007/s10766-006-0012-3}
}
@inproceedings{vasilache2006polyhedral,
	title        = {Polyhedral code generation in the real world},
	author       = {Vasilache, Nicolas and Bastoul, C{\'e}dric and Cohen, Albert},
	year         = 2006,
	booktitle    = {International Conference on Compiler Construction}
}
@mastersthesis{krizhevsky2009learning,
	title        = {Learning Multiple Layers of Features from Tiny Images},
	author       = {Krizhevsky, Alex},
	year         = 2009,
	school       = {University of Toronto}
}
@inproceedings{verdoolaege2010isl,
	title        = {isl: An integer set library for the polyhedral model},
	author       = {Verdoolaege, Sven},
	year         = 2010,
	booktitle    = {International Congress on Mathematical Software}
}
@inproceedings{grosser2011polly,
	title        = {Polly-Polyhedral optimization in LLVM},
	author       = {Grosser, Tobias and Zheng, Hongbin and Aloor, Raghesh and Simb{\"u}rger, Andreas and Gr{\"o}{\ss}linger, Armin and Pouchet, Louis-No{\"e}l},
	year         = 2011,
	booktitle    = {International Workshop on Polyhedral Compilation Techniques}
}
@inproceedings{chen2012benchnn,
	title        = {BenchNN: On the broad potential application scope of hardware neural network accelerators},
	author       = {Chen, Tianshi and Chen, Yunji and Duranton, Marc and Guo, Qi and Hashmi, Atif and Lipasti, Mikko and Nere, Andrew and Qiu, Shi and Sebag, Michele and Temam, Olivier},
	year         = 2012,
	booktitle    = {International Symposium on Workload Characterization}
}
@inproceedings{krizhevsky2012imagnet,
	title        = {Image{N}et classification with deep convolutional neural networks},
	author       = {Krizhevsky, Alex and Sutskever, Ilya and Hinton, Geoffrey},
	year         = 2012,
	booktitle    = {Advances in Neural Information Processing Systems},
	pages        = {1097--1105}
}
@inproceedings{denil2013predicting,
	title        = {Predicting parameters in deep learning},
	author       = {Denil, Misha and Shakibi, Babak and Dinh, Laurent and  Marc'Aurelio, Ranzato and de Freitas, Nando},
	year         = 2013,
	booktitle    = {Advances in Neural Information Processing Systems},
	pages        = {}
}
@inproceedings{ragan2013halide,
	title        = {Halide: a language and compiler for optimizing parallelism, locality, and recomputation in image processing pipelines},
	author       = {Ragan-Kelley, Jonathan and Barnes, Connelly and Adams, Andrew and Paris, Sylvain and Durand, Fr{\'e}do and Amarasinghe, Saman},
	year         = 2013,
	booktitle    = {Acm Sigplan Notices},
	volume       = 48,
	number       = 6,
	pages        = {519--530},
	organization = {ACM}
}
@article{verdoolaege2013polyhedral,
	title        = {Polyhedral parallel code generation for CUDA},
	author       = {Verdoolaege, Sven and Carlos Juega, Juan and Cohen, Albert and Ignacio Gomez, Jose and Tenllado, Christian and Catthoor, Francky},
	year         = 2013,
	journal      = {ACM Transactions on Architecture and Code Optimization (TACO)},
	publisher    = {ACM},
	volume       = 9,
	number       = 4,
	pages        = 54
}
@inproceedings{ba2014do,
	title        = {Do Deep Nets Really Need to be Deep?},
	author       = {Ba, Lei Jimmy and Caruana, Rich},
	year         = 2014,
	booktitle    = {Advances in Neural Information Processing Systems},
	pages        = {}
}
@inproceedings{lin2014microsoft,
	title        = {Microsoft {COCO}: Common objects in context},
	author       = {Lin, Tsung-Yi and Maire, Michael and Belongie, Serge and Hays, James and Perona, Pietro and Ramanan, Deva and Doll{\'a}r, Piotr and Zitnick, C. Lawrence},
	year         = 2014,
	booktitle    = {European Conference on Computer Vision},
	publisher    = {},
	address      = {},
	pages        = {}
}
@inproceedings{verdoolaege2014schedule,
	title        = {Schedule trees},
	author       = {Verdoolaege, Sven and Guelton, Serge and Grosser, Tobias and Cohen, Albert},
	year         = 2014,
	booktitle    = {International Workshop on Polyhedral Compilation Techniques}
}
@misc{chollet2015keras,
	title        = {Keras},
	author       = {Chollet, Fran{\c{c}}ois and others},
	year         = 2015
}
@article{hinton2015distilling,
	title        = {Distilling the knowledge in a neural network},
	author       = {Hinton, Geoffrey and Vinyals, Oriol and Dean, Jeff},
	year         = 2015,
	journal      = {arXiv preprint arXiv:1503.02531}
}
@inproceedings{ioffe2015batch,
	title        = {Batch normalization: Accelerating deep network training by reducing internal covariate shift},
	author       = {Ioffe, Sergey and Szegedy, Christian},
	year         = 2015,
	booktitle    = {International Conference on Machine Learning},
	publisher    = {Proceedings of Machine Learning Research},
	address      = {Stockholm, Sweden},
	pages        = {}
}
@inproceedings{srinivas2015data,
	title        = {Data-free parameter pruning for Deep Neural Networks},
	author       = {Suraj Srinivas and R. Venkatesh Babu},
	year         = 2015,
	booktitle    = {British Machine Vision Conference},
	publisher    = {BMVA Press},
	address      = {Swansea, UK},
	pages        = {}
}
@inproceedings{romero2015fitnets,
	title        = {Fit{N}ets: Hints for thin deep nets},
	author       = {Romero, Adriana and Ballas, Nicolas and Kahou, Samira Ebrahimi and Chassang, Antoine and Gatta, Carlo and Bengio, Yoshua},
	year         = 2015,
	booktitle    = {International Conference on Learning Representations},
	publisher    = {},
	pages        = {}
}
@article{russakovsky2015imagenet,
	title        = {Image{N}et Large Scale Visual Recognition Challenge},
	author       = {Olga Russakovsky and Jia Deng and Hao Su and Jonathan Krause and Sanjeev Satheesh and Sean Ma and Zhiheng Huang and Andrej Karpathy and Aditya Khosla and Michael Bernstein and Alexander C. Berg and Li Fei-Fei},
	year         = 2015,
	journal      = {Int. Journal of Computer Vision (IJCV)},
	volume       = 115,
	number       = 3,
	pages        = {211--252},
	doi          = {10.1007/s11263-015-0816-y}
}
@inproceedings{abadi2016tensorflow,
	title        = {Tensorflow: A system for large-scale machine learning},
	author       = {Abadi, Mart{\'\i}n and Barham, Paul and Chen, Jianmin and Chen, Zhifeng and Davis, Andy and Dean, Jeffrey and Devin, Matthieu and Ghemawat, Sanjay and Irving, Geoffrey and Isard, Michael and others},
	year         = 2016,
	booktitle    = {{USENIX} Symposium on Operating Systems Design and Implementation}
}
@inproceedings{chen2016xgboost,
	title        = {Xgboost: A scalable tree boosting system},
	author       = {Chen, Tianqi and Guestrin, Carlos},
	year         = 2016,
	booktitle    = {Proceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining},
	pages        = {785--794},
	organization = {ACM}
}
@inproceedings{guo2016dynamic,
	title        = {Dynamic Network Surgery for Efficient {DNNs}},
	author       = {Guo, Yiwen and Yao, Anbang and Chen, Yurong},
	year         = 2016,
	booktitle    = {Advances in Neural Information Processing Systems},
	pages        = {}
}
@inproceedings{han2016deep,
	title        = {Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding},
	author       = {Han, Song and Mao, Huizi and Dally, William J.},
	year         = 2016,
	booktitle    = {International Conference on Learning Representations},
	pages        = {}
}
@incollection{bottou2010large,
	title        = {Large-scale machine learning with stochastic gradient descent},
	author       = {Bottou, L{\'e}on},
	year         = 2010,
	booktitle    = {Proceedings of COMPSTAT'2010},
	publisher    = {Springer},
	pages        = {177--186}
}
@inproceedings{he2016deep,
	title        = {Deep residual learning for image recognition},
	author       = {He, Kaiming and Zhang, Xiangyu and Ren, Shaoqing and Sun, Jian},
	year         = 2016,
	booktitle    = {Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition},
	pages        = {}
}
@inproceedings{zagoruyko2016wide,
	title        = {Wide Residual Networks},
	author       = {Zagoruyko, Sergey  and Komodakis, Nikos},
	year         = 2016,
	booktitle    = {British Machine Vision Conference},
	publisher    = {},
	address      = {},
	pages        = {}
}
@inproceedings{chollet2017xception,
	title        = {{Xception}: Deep Learning with Depthwise Separable Convolutions},
	author       = {Fran{\c{c}}ois Chollet},
	year         = 2017,
	booktitle    = {Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition}
}
@inproceedings{choi2017towards,
	title        = {Towards the Limit of Network Quantization},
	author       = {Yoojin Choi and Mostafa El{-}Khamy and Jungwon Lee},
	year         = 2017,
	booktitle    = {International Conference on Learning Representations},
	publisher    = {},
	pages        = {}
}
@misc{devries2017improved,
	title        = {Improved regularization of convolutional neural networks with cutout},
	author       = {DeVries, Terrance and Taylor, Graham W.},
	year         = 2017,
	journal      = {arXiv preprint arXiv:1708.04552},
	publisher    = {},
	address      = {},
	pages        = {}
}
@inproceedings{he2017mask,
	title        = {Mask {R-CNN}},
	author       = {He, Kaiming and Gkioxari, Georgia and Doll{\'a}r, Piotr and Girshick, Ross},
	year         = 2017,
	booktitle    = {Proceedings of the IEEE International Conference on Computer Vision},
	publisher    = {},
	address      = {},
	pages        = {}
}
@inproceedings{hou2017loss,
	title        = {Loss-aware Binarization of Deep Networks},
	author       = {Lu Hou and Quanming Yao and James T. Kwok},
	year         = 2017,
	booktitle    = {International Conference on Learning Representations},
	publisher    = {},
	pages        = {}
}
@article{howard2017mobilenets,
	title        = {Mobile{N}ets: Efficient convolutional neural networks for mobile vision applications},
	author       = {Howard, Andrew G. and Zhu, Menglong and Chen, Bo and Kalenichenko, Dmitry and Wang, Weijun and Weyand, Tobias and Andreetto, Marco and Adam, Hartwig},
	year         = 2017,
	journal      = {arXiv preprint arXiv:1704.04861}
}
@inproceedings{ioannou2017deep,
	title        = {Deep Roots: Improving {CNN} Efficiency with Hierarchical Filter Groups},
	author       = {Ioannou, Yani and Robertson, Duncan and Cipolla, Roberto and Criminisi, Antonio},
	year         = 2017,
	booktitle    = {Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition}
}
@inproceedings{molchanov2017pruning,
	title        = {Pruning Convolutional Neural Networks for Resource Efficient Inference},
	author       = {Molchanov, Pavlo and Tyree, Stephen and Karras, Tero and Aila, Timo and Kautz, Jan},
	year         = 2017,
	booktitle    = {International Conference on Learning Representations},
	publisher    = {},
	pages        = {}
}
@article{paszke2017pytorch,
	title        = {Pytorch: Tensors and dynamic neural networks in python with strong gpu acceleration},
	author       = {Paszke, Adam and Gross, Sam and Chintala, Soumith and Chanan, Gregory},
	year         = 2017,
	journal      = {PyTorch: Tensors and dynamic neural networks in Python with strong GPU acceleration},
	volume       = 6
}
@inproceedings{steuwer2017lift,
	title        = {Lift: a functional data-parallel IR for high-performance GPU code generation},
	author       = {Steuwer, Michel and Remmelg, Toomas and Dubach, Christophe},
	year         = 2017,
	booktitle    = {International Symposium on Code Generation and Optimization}
}
@article{sze2017efficient,
	title        = {Efficient processing of deep neural networks: A tutorial and survey},
	author       = {Sze, Vivienne and Chen, Yu-Hsin and Yang, Tien-Ju and Emer, Joel S},
	year         = 2017,
	journal      = {Proceedings of the IEEE},
	publisher    = {Ieee},
	volume       = 105,
	number       = 12,
	pages        = {2295--2329}
}
@inproceedings{urban2017deep,
	title        = {Do deep convolutional nets really need to be deep and convolutional?},
	author       = {Urban, Gregor and Geras, Krzysztof J. and Kahou, Samira Ebrahimi and Aslan, Ozlem and Wang, Shengjie and Caruana, Rich and Mohamed, Abdelrahman and Philipose, Matthai and Richardson, Matt},
	year         = 2017,
	booktitle    = {International Conference on Learning Representations},
	publisher    = {},
	pages        = {}
}
@inproceedings{xie2017aggregated,
	title        = {Aggregated Residual Transformations for Deep Neural Networks},
	author       = {Xie, Saining and Girshick, Ross and Doll{\'a}r, Piotr and Tu, Zhuowen and He, Kaiming},
	year         = 2017,
	booktitle    = {Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition}
}
@inproceedings{zagoruyko2017paying,
	title        = {Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfer},
	author       = {Zagoruyko, Sergey  and Komodakis, Nikos},
	year         = 2017,
	booktitle    = {International Conference on Learning Representations},
	publisher    = {},
	pages        = {}
}
@misc{zhu2017prune,
	title        = {To prune, or not to prune: exploring the efficacy of pruning for model compression},
	author       = {Zhu, Michael and Gupta, Suyog},
	year         = 2017,
	journal      = {arXiv preprint arXiv:1710.01878}
}
@inproceedings{zoph2017neural,
	title        = {Neural architecture search with reinforcement learning},
	author       = {Zoph, Barret and Le, Quoc V.},
	year         = 2017,
	booktitle    = {International Conference on Learning Representations},
	publisher    = {},
	pages        = {}
}
@inproceedings{chen2018searching,
	title        = {Searching for efficient multi-scale architectures for dense image prediction},
	author       = {Chen, Liang-Chieh and Collins, Maxwell and Zhu, Yukun and Papandreou, George and Zoph, Barret and Schroff, Florian and Adam, Hartwig and Shlens, Jon},
	year         = 2018,
	booktitle    = {Advances in Neural Information Processing Systems},
	pages        = {}
}
@inproceedings{chen2018tvm,
	title        = {$\{$TVM$\}$: An automated end-to-end optimizing compiler for deep learning},
	author       = {Chen, Tianqi and Moreau, Thierry and Jiang, Ziheng and Zheng, Lianmin and Yan, Eddie and Shen, Haichen and Cowan, Meghan and Wang, Leyuan and Hu, Yuwei and Ceze, Luis and others},
	year         = 2018,
	booktitle    = {{USENIX} Symposium on Operating Systems Design and Implementation}
}
@article{crowley2018closer,
	title        = {A Closer Look at Structured Pruning for Neural Network Compression},
	author       = {Crowley, Elliot J and Turner, Jack and Storkey, Amos and O'Boyle, Michael},
	year         = 2018,
	journal      = {arXiv preprint arXiv:1810.04622}
}
@inproceedings{crowley2018moonshine,
	title        = {Moonshine: Distilling with Cheap Convolutions},
	author       = {Crowley, Elliot J. and Gray, Gavin and Storkey, Amos},
	year         = 2018,
	booktitle    = {Advances in Neural Information Processing Systems},
	pages        = {}
}
@inproceedings{furlanello2018born,
	title        = {Born again neural networks},
	author       = {Furlanello, Tommaso and Lipton, Zachary C. and Tschannen, Michael and Itti, Laurent and Anandkumar, Anima},
	year         = 2018,
	booktitle    = {International Conference on Machine Learning},
	publisher    = {Proceedings of Machine Learning Research},
	address      = {Stockholm, Sweden}
}
@inproceedings{gao2018condensenet,
	title        = {Condense{N}et: An Efficient DenseNet using Learned Group Convolutions},
	author       = {Huang, Gao and Liu, Schichen and van~der~Maaten, Laurens and Weinberger, Kilian~Q.},
	year         = 2018,
	booktitle    = {Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition},
	pages        = {}
}
@inproceedings{he2018amc,
	title        = {{AMC}: Automl for model compression and acceleration on mobile devices},
	author       = {He, Yihui and Lin, Ji and Liu, Zhijian and Wang, Hanrui and Li, Li-Jia and Han, Song},
	year         = 2018,
	booktitle    = {Proceedings of the European Conference on Computer Vision (ECCV)}
}
@inproceedings{hu2018squeeze,
	title        = {Squeeze-and-excitation networks},
	author       = {Hu, Jie and Shen, Li and Sun, Gang},
	year         = 2018,
	booktitle    = {Proceedings of the IEEE conference on computer vision and pattern recognition}
}
@inproceedings{kim2018paraphrasing,
	title        = {Paraphrasing Complex Network: Network Compression via Factor Transfer},
	author       = {Kim, Jangho and Park, Seonguk and Kwak, Nojun},
	year         = 2018,
	booktitle    = {Advances in Neural Information Processing Systems},
	pages        = {}
}
@inproceedings{luo2018neural,
	title        = {Neural architecture optimization},
	author       = {Luo, Renqian and Tian, Fei and Qin, Tao and Chen, Enhong and Liu, Tie-Yan},
	year         = 2018,
	booktitle    = {Advances in Neural Information Processing Systems}
}
@article{peng2018accelerating,
	title        = {Accelerating Deep Neural Networks with Spatial Bottleneck Modules},
	author       = {Peng, Junran and Xie, Lingxi and Zhang, Zhaoxiang and Tan, Tieniu and Wang, Jingdong},
	year         = 2018,
	journal      = {arXiv preprint arXiv:1809.02601},
	publisher    = {},
	address      = {},
	pages        = {}
}
@inproceedings{pham2018efficient,
	title        = {Efficient neural architecture search via parameter sharing},
	author       = {Pham, Hieu and Guan, Melody Y. and Zoph, Barret and Le, Quoc V. and Dean, Jeff},
	year         = 2018,
	booktitle    = {International Conference on Machine Learning},
	pages        = {}
}
@inproceedings{sandler2018mobilenetv2,
	title        = {Mobile{N}etv2: Inverted residuals and linear bottlenecks},
	author       = {Sandler, Mark and Howard, Andrew and Zhu, Menglong and Zhmoginov, Andrey and Chen, Liang-Chieh},
	year         = 2018,
	booktitle    = {Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition}
}
@misc{theis2018faster,
	title        = {Faster gaze prediction with dense networks and {F}isher pruning},
	author       = {Theis, Lucas and Korshunova, Iryna and Tejani, Alykhan and Husz{\'a}r, Ferenc},
	year         = 2018,
	journal      = {arXiv preprint arXiv:1801.05787}
}
@inproceedings{wu2018shift,
	title        = {Shift: A zero flop, zero parameter alternative to spatial convolutions},
	author       = {Wu, Bichen and Wan, Alvin and Yue, Xiangyu and Jin, Peter and Zhao, Sicheng and Golmant, Noah and Gholaminejad, Amir and Gonzalez, Joseph and Keutzer, Kurt},
	year         = 2018,
	booktitle    = {Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition}
}
@inproceedings{yang2018netadapt,
	title        = {Netadapt: Platform-aware neural network adaptation for mobile applications},
	author       = {Yang, Tien-Ju and Howard, Andrew and Chen, Bo and Zhang, Xiao and Go, Alec and Sandler, Mark and Sze, Vivienne and Adam, Hartwig},
	year         = 2018,
	booktitle    = {Proceedings of the European Conference on Computer Vision (ECCV)}
}
@inproceedings{zhang2018efficient,
	title        = {Efficient semantic scene completion network with spatial group convolution},
	author       = {Zhang, Jiahui and Zhao, Hao and Yao, Anbang and Chen, Yurong and Zhang, Li and Liao, Hongen},
	year         = 2018,
	booktitle    = {Proceedings of the European Conference on Computer Vision (ECCV)}
}
@inproceedings{zhang2018shufflenet,
	title        = {Shuffle{N}et: An extremely efficient convolutional neural network for mobile devices},
	author       = {Zhang, Xiangyu and Zhou, Xinyu and Lin, Mengxiao and Sun, Jian},
	year         = 2018,
	booktitle    = {Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition}
}
@inproceedings{zoph2018learning,
	title        = {Learning transferable architectures for scalable image recognition},
	author       = {Zoph, Barret and Vasudevan, Vijay and Shlens, Jonathon and Le, Quoc V.},
	year         = 2018,
	booktitle    = {Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition}
}
@inproceedings{barham2019machine,
	title        = {Machine Learning Systems are Stuck in a Rut},
	author       = {Barham, Paul and Isard, Michael},
	year         = 2019,
	booktitle    = {Proceedings of the Workshop on Hot Topics in Operating Systems},
	pages        = {177--183},
	organization = {ACM}
}
@article{decker2019python,
	title        = {The 800 Pound Python in the Machine Learning Room},
	author       = {Decker, James M and Moldovan, Dan and Wei, Guannan and Bhardwaj, Vritant and Essertel, Gregory and Wang, Fei and Wiltschko, Alexander B and Rompf, Tiark},
	journal      = {ret},
	volume       = 10,
	pages        = {0}
}
@article{gray2019separable,
	title        = {Separable Layers Enable Structured Efficient Linear Substitutions},
	author       = {Gray, Gavin and Crowley, Elliot J and Storkey, Amos},
	year         = 2019,
	journal      = {arXiv preprint arXiv:1906.00859}
}
@inproceedings{golub2019fulldeep,
	title        = {Full Deep Neural Network Training On a Pruned Weight Budget},
	author       = {Golub, Maximilian and Lemieux, Guy and Lis, Mieszko},
	year         = 2019,
	booktitle    = {Proceedings of the SysML Conference}
}
@inproceedings{frankle2019lottery,
	title        = {The lottery ticket hypothesis: Finding sparse, trainable neural networks},
	author       = {Frankle, Jonathan and Carbin, Michael},
	year         = 2019,
	booktitle    = {International Conference on Learning Representations},
	publisher    = {},
	pages        = {}
}
@inproceedings{jia2019taso,
	title        = {TASO: Optimizing Deep Learning Computation with Automatic Generation of Graph Substitutions},
	author       = {Jia, Zhihao and Padon, Oded and Thomas, James and Warszawski, Todd and Zaharia, Matei and Aiken, Alex},
	year         = 2019,
	booktitle    = {Symposium on Operating Systems Principles}
}
@article{lattner2019mlir,
	title        = {MLIR Primer: A Compiler Infrastructure for the End of Moore’s Law},
	author       = {Lattner, Chris and Pienaar, Jacques},
	year         = 2019
}
@inproceedings{lee2019snip,
	title        = {{SNIP}: Single-shot network pruning based on connection sensitivity},
	author       = {Lee, Namhoon and Ajanthan, Thalaiyasingam and Torr, Philip H. S.},
	year         = 2019,
	booktitle    = {International Conference on Learning Representations},
	publisher    = {},
	pages        = {}
}
@article{li2019random,
	title        = {Random search and reproducibility for neural architecture search},
	author       = {Li, Liam and Talwalkar, Ameet},
	year         = 2019,
	journal      = {arXiv preprint arXiv:1902.07638}
}
@inproceedings{liu2019darts,
	title        = {{DARTS}: Differentiable Architecture Search},
	author       = {Liu, Hanxiao and Simonyan, Karen and Yang, Yiming},
	year         = 2019,
	booktitle    = {International Conference on Learning Representations},
	publisher    = {},
	pages        = {}
}
@inproceedings{liu2019rethinking,
	title        = {Rethinking the value of network pruning},
	author       = {Liu, Zhuang and Sun, Mingjie and Zhou, Tinghui and Huang, Gao and Darrell, Trevor},
	year         = 2019,
	booktitle    = {International Conference on Learning Representations},
	publisher    = {},
	pages        = {}
}
@article{sciuto2019evaluating,
	title        = {Evaluating the Search Phase of Neural Architecture Search},
	author       = {Sciuto, Christian and Yu, Kaicheng and Jaggi, Martin and Musat, Claudiu and Salzmann, Mathieu},
	year         = 2019,
	journal      = {arXiv preprint arXiv:1902.08142}
}
@inproceedings{tan2019efficientnet,
	title        = {EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks},
	author       = {Tan, Mingxing and Le, Quoc V.},
	year         = 2019,
	booktitle    = {International Conference on Machine Learning}
}
@inproceedings{tan2019mnasnet,
	title        = {Mnas{N}et: Platform-aware neural architecture search for mobile},
	author       = {Tan, Mingxing and Chen, Bo and Pang, Ruoming and Vasudevan, Vijay and Sandler, Mark and Howard, Andrew and Le, Quoc V.},
	year         = 2019,
	booktitle    = {Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition}
}
@inproceedings{tillet2019triton,
	title        = {Triton: an intermediate language and compiler for tiled neural network computations},
	author       = {Tillet, Philippe and Kung, HT and Cox, David},
	year         = 2019,
	booktitle    = {International Workshop on Machine Learning and Programming Languages}
}
@article{wu2019convolution,
	title        = {Convolution with even-sized kernels and symmetric padding},
	author       = {Wu, Shuang and Wang, Guanrui and Tang, Pei and Chen, Feng and Shi, Luping},
	year         = 2019,
	journal      = {arXiv preprint arXiv:1903.08385}
}
@inproceedings{wu2019fbnet,
	title        = {{FBN}et: Hardware-aware efficient convnet design via differentiable neural architecture search},
	author       = {Wu, Bichen and Dai, Xiaoliang and Zhang, Peizhao and Wang, Yanghan and Sun, Fei and Wu, Yiming and Tian, Yuandong and Vajda, Peter and Jia, Yangqing and Keutzer, Kurt},
	year         = 2019,
	booktitle    = {Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition}
}
@article{zerrell2019stripe,
	title        = {Stripe: Tensor Compilation via the Nested Polyhedral Model},
	author       = {Zerrell, Tim and Bruestle, Jeremy},
	year         = 2019,
	journal      = {arXiv preprint arXiv:1903.06498}
}
@inproceedings{Turner2020BlockSwap:,
	title        = {BlockSwap: Fisher-guided Block Substitution for Network Compression on a Budget},
	author       = {Jack Turner and Elliot J. Crowley and Michael O'Boyle and Amos Storkey and Gavin Gray},
	year         = 2020,
	booktitle    = {International Conference on Learning Representations},
	url          = {https://openreview.net/forum?id=SklkDkSFPB}
}
@misc{torchvision,
	title        = {Torchvision},
	note         = {Accessed: 2019-11-15},
	howpublished = {\url{https://github.com/pytorch/vision}}
}
@article{schmidhuber2015deep,
	title        = {Deep learning in neural networks: An overview},
	author       = {Schmidhuber, J{\"u}rgen},
	year         = 2015,
	journal      = {Neural networks},
	publisher    = {Elsevier},
	volume       = 61,
	pages        = {85--117}
}
@article{lecun2015deep,
	title        = {Deep learning},
	author       = {LeCun, Yann and Bengio, Yoshua and Hinton, Geoffrey},
	year         = 2015,
	journal      = {Nature},
	publisher    = {Nature Publishing Group},
	volume       = 521,
	number       = 7553,
	pages        = 436
}
@article{strubell2019energy,
	title        = {Energy and Policy Considerations for Deep Learning in NLP},
	author       = {Strubell, Emma and Ganesh, Ananya and McCallum, Andrew},
	year         = 2019,
	journal      = {arXiv preprint arXiv:1906.02243}
}
@article{schwartz2019green,
	title        = {Green ai},
	author       = {Schwartz, Roy and Dodge, Jesse and Smith, Noah A and Etzioni, Oren},
	year         = 2019,
	journal      = {arXiv preprint arXiv:1907.10597}
}
@inproceedings{cai2019proxylessnas,
	title        = {Proxyless{NAS}: Direct neural architecture search on target task and hardware},
	author       = {Cai, Han and Zhu, Ligeng and Han, Song},
	year         = 2019,
	booktitle    = {International Conference on Learning Representations},
	publisher    = {}
}
@inproceedings{yang2020towards,
	title        = {Towards fairer datasets: Filtering and balancing the distribution of the people subtree in the imagenet hierarchy},
	author       = {Yang, Kaiyu and Qinami, Klint and Fei-Fei, Li and Deng, Jia and Russakovsky, Olga},
	year         = 2020,
	booktitle    = {Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency},
	pages        = {547--558}
}
@article{chowdhery2019visual,
	title        = {Visual Wake Words Dataset},
	author       = {Chowdhery, Aakanksha and Warden, Pete and Shlens, Jonathon and Howard, Andrew and Rhodes, Rocky},
	year         = 2019,
	journal      = {arXiv preprint arXiv:1906.05721}
}
@inproceedings{huang2019gpipe,
	title        = {Gpipe: Efficient training of giant neural networks using pipeline parallelism},
	author       = {Huang, Yanping and Cheng, Youlong and Bapna, Ankur and Firat, Orhan and Chen, Dehao and Chen, Mia and Lee, HyoukJoong and Ngiam, Jiquan and Le, Quoc V and Wu, Yonghui and others},
	year         = 2019,
	booktitle    = {Advances in Neural Information Processing Systems}
}
@inproceedings{li2017pruning,
	title        = {Pruning filters for efficient convnets},
	author       = {Li, Hao and Kadav, Asim and Durdanovic, Igor and Samet, Hanan and Graf, Hans Peter},
	year         = 2017,
	booktitle    = {International Conference on Learning Representations},
	publisher    = {}
}
@article{gale2019state,
	title        = {The State of Sparsity in Deep Neural Networks},
	author       = {Gale, Trevor and Elsen, Erich and Hooker, Sara},
	year         = 2019,
	journal      = {arXiv preprint arXiv:1902.09574}
}
@article{ying2019bench,
	title        = {Nas-bench-101: Towards reproducible neural architecture search},
	author       = {Ying, Chris and Klein, Aaron and Real, Esteban and Christiansen, Eric and Murphy, Kevin and Hutter, Frank},
	year         = 2019,
	journal      = {arXiv preprint arXiv:1902.09635}
}
@article{zela2020bench,
	title        = {NAS-Bench-1Shot1: Benchmarking and Dissecting One-shot Neural Architecture Search},
	author       = {Zela, Arber and Siems, Julien and Hutter, Frank},
	year         = 2020,
	journal      = {arXiv preprint arXiv:2001.10422}
}
@inproceedings{Dong2020NAS-Bench-201:,
	title        = {NAS-Bench-201: Extending the Scope of Reproducible Neural Architecture Search},
	author       = {Xuanyi Dong and Yi Yang},
	year         = 2020,
	booktitle    = {International Conference on Learning Representations},
	url          = {https://openreview.net/forum?id=HJxyZkBKDr}
}
@article{wistuba2019survey,
	title        = {A survey on neural architecture search},
	author       = {Wistuba, Martin and Rawat, Ambrish and Pedapati, Tejaswini},
	year         = 2019,
	journal      = {arXiv preprint arXiv:1905.01392}
}
@inproceedings{huang2017densely,
	title        = {Densely connected convolutional networks},
	author       = {Huang, Gao and Liu, Zhuang and Van Der Maaten, Laurens and Weinberger, Kilian Q},
	year         = 2017,
	booktitle    = {Proceedings of the IEEE conference on computer vision and pattern recognition},
	pages        = {4700--4708}
}

main.bbl
%%% -*-BibTeX-*-
%%% Do NOT edit. File created by BibTeX with style
%%% ACM-Reference-Format-Journals [18-Jan-2012].

\begin{thebibliography}{86}

%%% ====================================================================
%%% NOTE TO THE USER: you can override these defaults by providing
%%% customized versions of any of these macros before the \bibliography
%%% command.  Each of them MUST provide its own final punctuation,
%%% except for \shownote{}, \showDOI{}, and \showURL{}.  The latter two
%%% do not use final punctuation, in order to avoid confusing it with
%%% the Web address.
%%%
%%% To suppress output of a particular field, define its macro to expand
%%% to an empty string, or better, \unskip, like this:
%%%
%%% \newcommand{\showDOI}[1]{\unskip}   % LaTeX syntax
%%%
%%% \def \showDOI #1{\unskip}           % plain TeX syntax
%%%
%%% ====================================================================

\ifx \showCODEN    \undefined \def \showCODEN     #1{\unskip}     \fi
\ifx \showDOI      \undefined \def \showDOI       #1{#1}\fi
\ifx \showISBNx    \undefined \def \showISBNx     #1{\unskip}     \fi
\ifx \showISBNxiii \undefined \def \showISBNxiii  #1{\unskip}     \fi
\ifx \showISSN     \undefined \def \showISSN      #1{\unskip}     \fi
\ifx \showLCCN     \undefined \def \showLCCN      #1{\unskip}     \fi
\ifx \shownote     \undefined \def \shownote      #1{#1}          \fi
\ifx \showarticletitle \undefined \def \showarticletitle #1{#1}   \fi
\ifx \showURL      \undefined \def \showURL       {\relax}        \fi
% The following commands are used for tagged output and should be
% invisible to TeX
\providecommand\bibfield[2]{#2}
\providecommand\bibinfo[2]{#2}
\providecommand\natexlab[1]{#1}
\providecommand\showeprint[2][]{arXiv:#2}

\bibitem[\protect\citeauthoryear{Abadi, Barham, Chen, Chen, Davis, Dean, Devin,
  Ghemawat, Irving, Isard, et~al\mbox{.}}{Abadi et~al\mbox{.}}{2016}]%
        {abadi2016tensorflow}
\bibfield{author}{\bibinfo{person}{Mart{\'\i}n Abadi}, \bibinfo{person}{Paul
  Barham}, \bibinfo{person}{Jianmin Chen}, \bibinfo{person}{Zhifeng Chen},
  \bibinfo{person}{Andy Davis}, \bibinfo{person}{Jeffrey Dean},
  \bibinfo{person}{Matthieu Devin}, \bibinfo{person}{Sanjay Ghemawat},
  \bibinfo{person}{Geoffrey Irving}, \bibinfo{person}{Michael Isard},
  {et~al\mbox{.}}} \bibinfo{year}{2016}\natexlab{}.
\newblock \showarticletitle{Tensorflow: A system for large-scale machine
  learning}. In \bibinfo{booktitle}{\emph{{USENIX} Symposium on Operating
  Systems Design and Implementation}}.
\newblock


\bibitem[\protect\citeauthoryear{Abdelfattah, Mehrotra, Dudziak, and
  Lane}{Abdelfattah et~al\mbox{.}}{2021}]%
        {abdelfattah2021zero}
\bibfield{author}{\bibinfo{person}{Mohamed~S Abdelfattah},
  \bibinfo{person}{Abhinav Mehrotra}, \bibinfo{person}{{\L}ukasz Dudziak},
  {and} \bibinfo{person}{Nicholas~D Lane}.} \bibinfo{year}{2021}\natexlab{}.
\newblock \showarticletitle{Zero-Cost Proxies for Lightweight NAS}.
\newblock \bibinfo{journal}{\emph{arXiv preprint arXiv:2101.08134}}
  (\bibinfo{year}{2021}).
\newblock


\bibitem[\protect\citeauthoryear{Ba and Caruana}{Ba and Caruana}{2014}]%
        {ba2014do}
\bibfield{author}{\bibinfo{person}{Lei~Jimmy Ba} {and} \bibinfo{person}{Rich
  Caruana}.} \bibinfo{year}{2014}\natexlab{}.
\newblock \showarticletitle{Do Deep Nets Really Need to be Deep?}. In
  \bibinfo{booktitle}{\emph{Advances in Neural Information Processing
  Systems}}.
\newblock


\bibitem[\protect\citeauthoryear{Barham and Isard}{Barham and Isard}{2019}]%
        {barham2019machine}
\bibfield{author}{\bibinfo{person}{Paul Barham} {and} \bibinfo{person}{Michael
  Isard}.} \bibinfo{year}{2019}\natexlab{}.
\newblock \showarticletitle{Machine Learning Systems are Stuck in a Rut}. In
  \bibinfo{booktitle}{\emph{Proceedings of the Workshop on Hot Topics in
  Operating Systems}}. ACM, \bibinfo{pages}{177--183}.
\newblock


\bibitem[\protect\citeauthoryear{Bondhugula, Hartono, Ramanujam, and
  Sadayappan}{Bondhugula et~al\mbox{.}}{2008}]%
        {uday08}
\bibfield{author}{\bibinfo{person}{Uday Bondhugula}, \bibinfo{person}{Albert
  Hartono}, \bibinfo{person}{J. Ramanujam}, {and} \bibinfo{person}{P.
  Sadayappan}.} \bibinfo{year}{2008}\natexlab{}.
\newblock \showarticletitle{A Practical Automatic Polyhedral Parallelizer and
  Locality Optimizer}. In \bibinfo{booktitle}{\emph{Proceedings of the 29th ACM
  SIGPLAN Conference on Programming Language Design and Implementation}}
  (Tucson, AZ, USA) \emph{(\bibinfo{series}{PLDI ’08})}.
  \bibinfo{publisher}{Association for Computing Machinery},
  \bibinfo{address}{New York, NY, USA}, \bibinfo{pages}{101–113}.
\newblock
\showISBNx{9781595938602}
\urldef\tempurl%
\url{https://doi.org/10.1145/1375581.1375595}
\showDOI{\tempurl}


\bibitem[\protect\citeauthoryear{Bottou}{Bottou}{2010}]%
        {bottou2010large}
\bibfield{author}{\bibinfo{person}{L{\'e}on Bottou}.}
  \bibinfo{year}{2010}\natexlab{}.
\newblock \showarticletitle{Large-scale machine learning with stochastic
  gradient descent}.
\newblock In \bibinfo{booktitle}{\emph{Proceedings of COMPSTAT'2010}}.
  \bibinfo{publisher}{Springer}, \bibinfo{pages}{177--186}.
\newblock


\bibitem[\protect\citeauthoryear{Chen, Collins, Zhu, Papandreou, Zoph, Schroff,
  Adam, and Shlens}{Chen et~al\mbox{.}}{2018a}]%
        {chen2018searching}
\bibfield{author}{\bibinfo{person}{Liang-Chieh Chen}, \bibinfo{person}{Maxwell
  Collins}, \bibinfo{person}{Yukun Zhu}, \bibinfo{person}{George Papandreou},
  \bibinfo{person}{Barret Zoph}, \bibinfo{person}{Florian Schroff},
  \bibinfo{person}{Hartwig Adam}, {and} \bibinfo{person}{Jon Shlens}.}
  \bibinfo{year}{2018}\natexlab{a}.
\newblock \showarticletitle{Searching for efficient multi-scale architectures
  for dense image prediction}. In \bibinfo{booktitle}{\emph{Advances in Neural
  Information Processing Systems}}.
\newblock


\bibitem[\protect\citeauthoryear{Chen, Chen, Duranton, Guo, Hashmi, Lipasti,
  Nere, Qiu, Sebag, and Temam}{Chen et~al\mbox{.}}{2012}]%
        {chen2012benchnn}
\bibfield{author}{\bibinfo{person}{Tianshi Chen}, \bibinfo{person}{Yunji Chen},
  \bibinfo{person}{Marc Duranton}, \bibinfo{person}{Qi Guo},
  \bibinfo{person}{Atif Hashmi}, \bibinfo{person}{Mikko Lipasti},
  \bibinfo{person}{Andrew Nere}, \bibinfo{person}{Shi Qiu},
  \bibinfo{person}{Michele Sebag}, {and} \bibinfo{person}{Olivier Temam}.}
  \bibinfo{year}{2012}\natexlab{}.
\newblock \showarticletitle{BenchNN: On the broad potential application scope
  of hardware neural network accelerators}. In
  \bibinfo{booktitle}{\emph{International Symposium on Workload
  Characterization}}.
\newblock


\bibitem[\protect\citeauthoryear{Chen, Du, Sun, Wang, Wu, Chen, and Temam}{Chen
  et~al\mbox{.}}{2014}]%
        {Chen:2014:DSH:2541940.2541967}
\bibfield{author}{\bibinfo{person}{Tianshi Chen}, \bibinfo{person}{Zidong Du},
  \bibinfo{person}{Ninghui Sun}, \bibinfo{person}{Jia Wang},
  \bibinfo{person}{Chengyong Wu}, \bibinfo{person}{Yunji Chen}, {and}
  \bibinfo{person}{Olivier Temam}.} \bibinfo{year}{2014}\natexlab{}.
\newblock \showarticletitle{DianNao: A Small-footprint High-throughput
  Accelerator for Ubiquitous Machine-learning}. In
  \bibinfo{booktitle}{\emph{Proceedings of the International Conference on
  Architectural Support for Programming Languages and Operating Systems}}.
\newblock


\bibitem[\protect\citeauthoryear{Chen, Moreau, Jiang, Zheng, Yan, Shen, Cowan,
  Wang, Hu, Ceze, et~al\mbox{.}}{Chen et~al\mbox{.}}{2018b}]%
        {chen2018tvm}
\bibfield{author}{\bibinfo{person}{Tianqi Chen}, \bibinfo{person}{Thierry
  Moreau}, \bibinfo{person}{Ziheng Jiang}, \bibinfo{person}{Lianmin Zheng},
  \bibinfo{person}{Eddie Yan}, \bibinfo{person}{Haichen Shen},
  \bibinfo{person}{Meghan Cowan}, \bibinfo{person}{Leyuan Wang},
  \bibinfo{person}{Yuwei Hu}, \bibinfo{person}{Luis Ceze}, {et~al\mbox{.}}}
  \bibinfo{year}{2018}\natexlab{b}.
\newblock \showarticletitle{$\{$TVM$\}$: An automated end-to-end optimizing
  compiler for deep learning}. In \bibinfo{booktitle}{\emph{{USENIX} Symposium
  on Operating Systems Design and Implementation}}.
\newblock


\bibitem[\protect\citeauthoryear{Chollet}{Chollet}{2017}]%
        {chollet2017xception}
\bibfield{author}{\bibinfo{person}{Fran{\c{c}}ois Chollet}.}
  \bibinfo{year}{2017}\natexlab{}.
\newblock \showarticletitle{{Xception}: Deep Learning with Depthwise Separable
  Convolutions}. In \bibinfo{booktitle}{\emph{Proceedings of the IEEE
  Conference on Computer Vision and Pattern Recognition}}.
\newblock


\bibitem[\protect\citeauthoryear{Chowdhery, Warden, Shlens, Howard, and
  Rhodes}{Chowdhery et~al\mbox{.}}{2019}]%
        {chowdhery2019visual}
\bibfield{author}{\bibinfo{person}{Aakanksha Chowdhery}, \bibinfo{person}{Pete
  Warden}, \bibinfo{person}{Jonathon Shlens}, \bibinfo{person}{Andrew Howard},
  {and} \bibinfo{person}{Rocky Rhodes}.} \bibinfo{year}{2019}\natexlab{}.
\newblock \showarticletitle{Visual Wake Words Dataset}.
\newblock \bibinfo{journal}{\emph{arXiv preprint arXiv:1906.05721}}
  (\bibinfo{year}{2019}).
\newblock


\bibitem[\protect\citeauthoryear{Cohen}{Cohen}{1999}]%
        {cohen1999analyse}
\bibfield{author}{\bibinfo{person}{Albert~Henri Cohen}.}
  \bibinfo{year}{1999}\natexlab{}.
\newblock \emph{\bibinfo{title}{Analyse et transformation de programmes: du
  mod{\`e}le poly{\'e}drique aux langages formels}}.
\newblock \bibinfo{thesistype}{Ph.D. Dissertation}.
  \bibinfo{school}{Versailles-St Quentin en Yvelines}.
\newblock


\bibitem[\protect\citeauthoryear{Crowley, Turner, Storkey, and O'Boyle}{Crowley
  et~al\mbox{.}}{2018}]%
        {crowley2018pruning}
\bibfield{author}{\bibinfo{person}{Elliot~J Crowley}, \bibinfo{person}{Jack
  Turner}, \bibinfo{person}{Amos Storkey}, {and} \bibinfo{person}{Michael
  O'Boyle}.} \bibinfo{year}{2018}\natexlab{}.
\newblock \showarticletitle{Pruning neural networks: is it time to nip it in
  the bud?}
\newblock \bibinfo{journal}{\emph{arXiv preprint arXiv:1810.04622}}
  (\bibinfo{year}{2018}).
\newblock


\bibitem[\protect\citeauthoryear{Cyphers, Bansal, Bhiwandiwalla, Bobba,
  Brookhart, Chakraborty, Constable, Convey, Cook, Kanawi, Kimball, Knight,
  Korovaiko, Kumar, Lao, Lishka, Menon, Myers, Narayana, Procter, and
  Webb}{Cyphers et~al\mbox{.}}{2018}]%
        {cyphers2018intel}
\bibfield{author}{\bibinfo{person}{Scott Cyphers}, \bibinfo{person}{Arjun~K.
  Bansal}, \bibinfo{person}{Anahita Bhiwandiwalla}, \bibinfo{person}{Jayaram
  Bobba}, \bibinfo{person}{Matthew Brookhart}, \bibinfo{person}{Avijit
  Chakraborty}, \bibinfo{person}{Will Constable}, \bibinfo{person}{Christian
  Convey}, \bibinfo{person}{Leona Cook}, \bibinfo{person}{Omar Kanawi},
  \bibinfo{person}{Robert Kimball}, \bibinfo{person}{Jason Knight},
  \bibinfo{person}{Nikolay Korovaiko}, \bibinfo{person}{Varun Kumar},
  \bibinfo{person}{Yixing Lao}, \bibinfo{person}{Christopher~R. Lishka},
  \bibinfo{person}{Jaikrishnan Menon}, \bibinfo{person}{Jennifer Myers},
  \bibinfo{person}{Sandeep~Aswath Narayana}, \bibinfo{person}{Adam Procter},
  {and} \bibinfo{person}{Tristan~J. Webb}.} \bibinfo{year}{2018}\natexlab{}.
\newblock \showarticletitle{Intel nGraph: An Intermediate Representation,
  Compiler, and Executor for Deep Learning}.
\newblock \bibinfo{journal}{\emph{arXiv preprint arXiv:1801.08058}}
  (\bibinfo{year}{2018}).
\newblock


\bibitem[\protect\citeauthoryear{Denil, Shakibi, Dinh, Marc'Aurelio, and
  de~Freitas}{Denil et~al\mbox{.}}{2013}]%
        {denil2013predicting}
\bibfield{author}{\bibinfo{person}{Misha Denil}, \bibinfo{person}{Babak
  Shakibi}, \bibinfo{person}{Laurent Dinh}, \bibinfo{person}{Ranzato
  Marc'Aurelio}, {and} \bibinfo{person}{Nando de Freitas}.}
  \bibinfo{year}{2013}\natexlab{}.
\newblock \showarticletitle{Predicting parameters in deep learning}. In
  \bibinfo{booktitle}{\emph{Advances in Neural Information Processing
  Systems}}.
\newblock


\bibitem[\protect\citeauthoryear{Dong and Yang}{Dong and Yang}{2020}]%
        {Dong2020NAS-Bench-201:}
\bibfield{author}{\bibinfo{person}{Xuanyi Dong} {and} \bibinfo{person}{Yi
  Yang}.} \bibinfo{year}{2020}\natexlab{}.
\newblock \showarticletitle{NAS-Bench-201: Extending the Scope of Reproducible
  Neural Architecture Search}. In \bibinfo{booktitle}{\emph{International
  Conference on Learning Representations}}.
\newblock
\urldef\tempurl%
\url{https://openreview.net/forum?id=HJxyZkBKDr}
\showURL{%
\tempurl}


\bibitem[\protect\citeauthoryear{Elango, Rubin, Ravishankar, Sandanagobalane,
  and Grover}{Elango et~al\mbox{.}}{2018}]%
        {Elango:2018:DDL:3211346.3211354}
\bibfield{author}{\bibinfo{person}{Venmugil Elango}, \bibinfo{person}{Norm
  Rubin}, \bibinfo{person}{Mahesh Ravishankar}, \bibinfo{person}{Hariharan
  Sandanagobalane}, {and} \bibinfo{person}{Vinod Grover}.}
  \bibinfo{year}{2018}\natexlab{}.
\newblock \showarticletitle{Diesel: DSL for Linear Algebra and Neural Net
  Computations on GPUs}. In \bibinfo{booktitle}{\emph{In International Workshop
  on Machine Learning and Programming Languages}}.
\newblock


\bibitem[\protect\citeauthoryear{Elsken, Metzen, Hutter, et~al\mbox{.}}{Elsken
  et~al\mbox{.}}{2019}]%
        {elsken2019neural}
\bibfield{author}{\bibinfo{person}{Thomas Elsken}, \bibinfo{person}{Jan~Hendrik
  Metzen}, \bibinfo{person}{Frank Hutter}, {et~al\mbox{.}}}
  \bibinfo{year}{2019}\natexlab{}.
\newblock \bibinfo{title}{Neural Architecture Search.}
\newblock
\newblock


\bibitem[\protect\citeauthoryear{Figurnov, Ibraimova, Vetrov, and
  Kohli}{Figurnov et~al\mbox{.}}{2015}]%
        {figurnov2015perforatedcnns}
\bibfield{author}{\bibinfo{person}{Michael Figurnov}, \bibinfo{person}{Aijan
  Ibraimova}, \bibinfo{person}{Dmitry Vetrov}, {and} \bibinfo{person}{Pushmeet
  Kohli}.} \bibinfo{year}{2015}\natexlab{}.
\newblock \showarticletitle{Perforatedcnns: Acceleration through elimination of
  redundant convolutions}.
\newblock \bibinfo{journal}{\emph{arXiv preprint arXiv:1504.08362}}
  (\bibinfo{year}{2015}).
\newblock


\bibitem[\protect\citeauthoryear{Frankle and Carbin}{Frankle and
  Carbin}{2019}]%
        {frankle2019lottery}
\bibfield{author}{\bibinfo{person}{Jonathan Frankle} {and}
  \bibinfo{person}{Michael Carbin}.} \bibinfo{year}{2019}\natexlab{}.
\newblock \showarticletitle{The lottery ticket hypothesis: Finding sparse,
  trainable neural networks}. In \bibinfo{booktitle}{\emph{International
  Conference on Learning Representations}}.
\newblock


\bibitem[\protect\citeauthoryear{Gale, Elsen, and Hooker}{Gale
  et~al\mbox{.}}{2019}]%
        {gale2019state}
\bibfield{author}{\bibinfo{person}{Trevor Gale}, \bibinfo{person}{Erich Elsen},
  {and} \bibinfo{person}{Sara Hooker}.} \bibinfo{year}{2019}\natexlab{}.
\newblock \showarticletitle{The State of Sparsity in Deep Neural Networks}.
\newblock \bibinfo{journal}{\emph{arXiv preprint arXiv:1902.09574}}
  (\bibinfo{year}{2019}).
\newblock


\bibitem[\protect\citeauthoryear{Gibson, Cano, Turner, Crowley, O’Boyle, and
  Storkey}{Gibson et~al\mbox{.}}{2020}]%
        {gibson2020optimizing}
\bibfield{author}{\bibinfo{person}{Perry Gibson}, \bibinfo{person}{Jos{\'e}
  Cano}, \bibinfo{person}{Jack Turner}, \bibinfo{person}{Elliot~J Crowley},
  \bibinfo{person}{Michael O’Boyle}, {and} \bibinfo{person}{Amos Storkey}.}
  \bibinfo{year}{2020}\natexlab{}.
\newblock \showarticletitle{Optimizing Grouped Convolutions on Edge Devices}.
  In \bibinfo{booktitle}{\emph{2020 IEEE 31st International Conference on
  Application-specific Systems, Architectures and Processors (ASAP)}}. IEEE,
  \bibinfo{pages}{189--196}.
\newblock


\bibitem[\protect\citeauthoryear{Girbal, Vasilache, Bastoul, Cohen, Parello,
  Sigler, and Temam}{Girbal et~al\mbox{.}}{2006}]%
        {Girbal2006}
\bibfield{author}{\bibinfo{person}{Sylvain Girbal}, \bibinfo{person}{Nicolas
  Vasilache}, \bibinfo{person}{C{\'e}dric Bastoul}, \bibinfo{person}{Albert
  Cohen}, \bibinfo{person}{David Parello}, \bibinfo{person}{Marc Sigler}, {and}
  \bibinfo{person}{Olivier Temam}.} \bibinfo{year}{2006}\natexlab{}.
\newblock \showarticletitle{Semi-Automatic Composition of Loop Transformations
  for Deep Parallelism and Memory Hierarchies}.
\newblock \bibinfo{journal}{\emph{International Journal of Parallel
  Programming}} \bibinfo{volume}{34}, \bibinfo{number}{3} (\bibinfo{date}{01
  June} \bibinfo{year}{2006}), \bibinfo{pages}{261--317}.
\newblock
\showISSN{1573-7640}
\urldef\tempurl%
\url{https://doi.org/10.1007/s10766-006-0012-3}
\showDOI{\tempurl}


\bibitem[\protect\citeauthoryear{Goiri, Bianchini, Nagarakatte, and
  Nguyen}{Goiri et~al\mbox{.}}{2015}]%
        {goiri2015approxhadoop}
\bibfield{author}{\bibinfo{person}{Inigo Goiri}, \bibinfo{person}{Ricardo
  Bianchini}, \bibinfo{person}{Santosh Nagarakatte}, {and}
  \bibinfo{person}{Thu~D Nguyen}.} \bibinfo{year}{2015}\natexlab{}.
\newblock \showarticletitle{Approxhadoop: Bringing approximations to mapreduce
  frameworks}. In \bibinfo{booktitle}{\emph{Proceedings of the Twentieth
  International Conference on Architectural Support for Programming Languages
  and Operating Systems}}. \bibinfo{pages}{383--397}.
\newblock


\bibitem[\protect\citeauthoryear{Golatkar, Achille, and Soatto}{Golatkar
  et~al\mbox{.}}{2019}]%
        {golatkar2019time}
\bibfield{author}{\bibinfo{person}{Aditya~Sharad Golatkar},
  \bibinfo{person}{Alessandro Achille}, {and} \bibinfo{person}{Stefano
  Soatto}.} \bibinfo{year}{2019}\natexlab{}.
\newblock \showarticletitle{Time matters in regularizing deep networks: Weight
  decay and data augmentation affect early learning dynamics, matter little
  near convergence}. In \bibinfo{booktitle}{\emph{Advances in Neural
  Information Processing Systems}}. \bibinfo{pages}{10678--10688}.
\newblock


\bibitem[\protect\citeauthoryear{Grosser, Zheng, Aloor, Simb{\"u}rger,
  Gr{\"o}{\ss}linger, and Pouchet}{Grosser et~al\mbox{.}}{2011}]%
        {grosser2011polly}
\bibfield{author}{\bibinfo{person}{Tobias Grosser}, \bibinfo{person}{Hongbin
  Zheng}, \bibinfo{person}{Raghesh Aloor}, \bibinfo{person}{Andreas
  Simb{\"u}rger}, \bibinfo{person}{Armin Gr{\"o}{\ss}linger}, {and}
  \bibinfo{person}{Louis-No{\"e}l Pouchet}.} \bibinfo{year}{2011}\natexlab{}.
\newblock \showarticletitle{Polly-Polyhedral optimization in LLVM}. In
  \bibinfo{booktitle}{\emph{International Workshop on Polyhedral Compilation
  Techniques}}.
\newblock


\bibitem[\protect\citeauthoryear{Han, Mao, and Dally}{Han
  et~al\mbox{.}}{2016}]%
        {han2016deep}
\bibfield{author}{\bibinfo{person}{Song Han}, \bibinfo{person}{Huizi Mao},
  {and} \bibinfo{person}{William~J. Dally}.} \bibinfo{year}{2016}\natexlab{}.
\newblock \showarticletitle{Deep compression: Compressing deep neural networks
  with pruning, trained quantization and huffman coding}. In
  \bibinfo{booktitle}{\emph{International Conference on Learning
  Representations}}.
\newblock


\bibitem[\protect\citeauthoryear{He, Zhang, Ren, and Sun}{He
  et~al\mbox{.}}{2016}]%
        {he2016deep}
\bibfield{author}{\bibinfo{person}{Kaiming He}, \bibinfo{person}{Xiangyu
  Zhang}, \bibinfo{person}{Shaoqing Ren}, {and} \bibinfo{person}{Jian Sun}.}
  \bibinfo{year}{2016}\natexlab{}.
\newblock \showarticletitle{Deep residual learning for image recognition}. In
  \bibinfo{booktitle}{\emph{Proceedings of the IEEE Conference on Computer
  Vision and Pattern Recognition}}.
\newblock


\bibitem[\protect\citeauthoryear{He, Lin, Liu, Wang, Li, and Han}{He
  et~al\mbox{.}}{2018}]%
        {he2018amc}
\bibfield{author}{\bibinfo{person}{Yihui He}, \bibinfo{person}{Ji Lin},
  \bibinfo{person}{Zhijian Liu}, \bibinfo{person}{Hanrui Wang},
  \bibinfo{person}{Li-Jia Li}, {and} \bibinfo{person}{Song Han}.}
  \bibinfo{year}{2018}\natexlab{}.
\newblock \showarticletitle{{AMC}: Automl for model compression and
  acceleration on mobile devices}. In \bibinfo{booktitle}{\emph{Proceedings of
  the European Conference on Computer Vision (ECCV)}}.
\newblock


\bibitem[\protect\citeauthoryear{Hinton, Vinyals, and Dean}{Hinton
  et~al\mbox{.}}{2015}]%
        {hinton2015distilling}
\bibfield{author}{\bibinfo{person}{Geoffrey Hinton}, \bibinfo{person}{Oriol
  Vinyals}, {and} \bibinfo{person}{Jeff Dean}.}
  \bibinfo{year}{2015}\natexlab{}.
\newblock \showarticletitle{Distilling the knowledge in a neural network}.
\newblock \bibinfo{journal}{\emph{arXiv preprint arXiv:1503.02531}}
  (\bibinfo{year}{2015}).
\newblock


\bibitem[\protect\citeauthoryear{Howard, Zhu, Chen, Kalenichenko, Wang, Weyand,
  Andreetto, and Adam}{Howard et~al\mbox{.}}{2017}]%
        {howard2017mobilenets}
\bibfield{author}{\bibinfo{person}{Andrew~G. Howard}, \bibinfo{person}{Menglong
  Zhu}, \bibinfo{person}{Bo Chen}, \bibinfo{person}{Dmitry Kalenichenko},
  \bibinfo{person}{Weijun Wang}, \bibinfo{person}{Tobias Weyand},
  \bibinfo{person}{Marco Andreetto}, {and} \bibinfo{person}{Hartwig Adam}.}
  \bibinfo{year}{2017}\natexlab{}.
\newblock \showarticletitle{Mobile{N}ets: Efficient convolutional neural
  networks for mobile vision applications}.
\newblock \bibinfo{journal}{\emph{arXiv preprint arXiv:1704.04861}}
  (\bibinfo{year}{2017}).
\newblock


\bibitem[\protect\citeauthoryear{Huang, Liu, van~der Maaten, and
  Weinberger}{Huang et~al\mbox{.}}{2018}]%
        {gao2018condensenet}
\bibfield{author}{\bibinfo{person}{Gao Huang}, \bibinfo{person}{Schichen Liu},
  \bibinfo{person}{Laurens van~der Maaten}, {and} \bibinfo{person}{Kilian~Q.
  Weinberger}.} \bibinfo{year}{2018}\natexlab{}.
\newblock \showarticletitle{Condense{N}et: An Efficient DenseNet using Learned
  Group Convolutions}. In \bibinfo{booktitle}{\emph{Proceedings of the IEEE
  Conference on Computer Vision and Pattern Recognition}}.
\newblock


\bibitem[\protect\citeauthoryear{Huang, Liu, Van Der~Maaten, and
  Weinberger}{Huang et~al\mbox{.}}{2017}]%
        {huang2017densely}
\bibfield{author}{\bibinfo{person}{Gao Huang}, \bibinfo{person}{Zhuang Liu},
  \bibinfo{person}{Laurens Van Der~Maaten}, {and} \bibinfo{person}{Kilian~Q
  Weinberger}.} \bibinfo{year}{2017}\natexlab{}.
\newblock \showarticletitle{Densely connected convolutional networks}. In
  \bibinfo{booktitle}{\emph{Proceedings of the IEEE conference on computer
  vision and pattern recognition}}. \bibinfo{pages}{4700--4708}.
\newblock


\bibitem[\protect\citeauthoryear{Huang, Cheng, Bapna, Firat, Chen, Chen, Lee,
  Ngiam, Le, Wu, et~al\mbox{.}}{Huang et~al\mbox{.}}{2019}]%
        {huang2019gpipe}
\bibfield{author}{\bibinfo{person}{Yanping Huang}, \bibinfo{person}{Youlong
  Cheng}, \bibinfo{person}{Ankur Bapna}, \bibinfo{person}{Orhan Firat},
  \bibinfo{person}{Dehao Chen}, \bibinfo{person}{Mia Chen},
  \bibinfo{person}{HyoukJoong Lee}, \bibinfo{person}{Jiquan Ngiam},
  \bibinfo{person}{Quoc~V Le}, \bibinfo{person}{Yonghui Wu}, {et~al\mbox{.}}}
  \bibinfo{year}{2019}\natexlab{}.
\newblock \showarticletitle{Gpipe: Efficient training of giant neural networks
  using pipeline parallelism}. In \bibinfo{booktitle}{\emph{Advances in Neural
  Information Processing Systems}}.
\newblock


\bibitem[\protect\citeauthoryear{Ioannou, Robertson, Cipolla, and
  Criminisi}{Ioannou et~al\mbox{.}}{2017}]%
        {ioannou2017deep}
\bibfield{author}{\bibinfo{person}{Yani Ioannou}, \bibinfo{person}{Duncan
  Robertson}, \bibinfo{person}{Roberto Cipolla}, {and} \bibinfo{person}{Antonio
  Criminisi}.} \bibinfo{year}{2017}\natexlab{}.
\newblock \showarticletitle{Deep Roots: Improving {CNN} Efficiency with
  Hierarchical Filter Groups}. In \bibinfo{booktitle}{\emph{Proceedings of the
  IEEE Conference on Computer Vision and Pattern Recognition}}.
\newblock


\bibitem[\protect\citeauthoryear{Jia, Padon, Thomas, Warszawski, Zaharia, and
  Aiken}{Jia et~al\mbox{.}}{2019}]%
        {jia2019taso}
\bibfield{author}{\bibinfo{person}{Zhihao Jia}, \bibinfo{person}{Oded Padon},
  \bibinfo{person}{James Thomas}, \bibinfo{person}{Todd Warszawski},
  \bibinfo{person}{Matei Zaharia}, {and} \bibinfo{person}{Alex Aiken}.}
  \bibinfo{year}{2019}\natexlab{}.
\newblock \showarticletitle{TASO: Optimizing Deep Learning Computation with
  Automatic Generation of Graph Substitutions}. In
  \bibinfo{booktitle}{\emph{Symposium on Operating Systems Principles}}.
\newblock


\bibitem[\protect\citeauthoryear{Kelly and Pugh}{Kelly and Pugh}{1998}]%
        {kelly1998framework}
\bibfield{author}{\bibinfo{person}{Wayne Kelly} {and} \bibinfo{person}{William
  Pugh}.} \bibinfo{year}{1998}\natexlab{}.
\newblock \bibinfo{booktitle}{\emph{A framework for unifying reordering
  transformations}}.
\newblock \bibinfo{type}{{T}echnical {R}eport}.
\newblock


\bibitem[\protect\citeauthoryear{LeCun, Bengio, and Hinton}{LeCun
  et~al\mbox{.}}{2015}]%
        {lecun2015deep}
\bibfield{author}{\bibinfo{person}{Yann LeCun}, \bibinfo{person}{Yoshua
  Bengio}, {and} \bibinfo{person}{Geoffrey Hinton}.}
  \bibinfo{year}{2015}\natexlab{}.
\newblock \showarticletitle{Deep learning}.
\newblock \bibinfo{journal}{\emph{Nature}} \bibinfo{volume}{521},
  \bibinfo{number}{7553} (\bibinfo{year}{2015}), \bibinfo{pages}{436}.
\newblock


\bibitem[\protect\citeauthoryear{LeCun, Denker, and Solla}{LeCun
  et~al\mbox{.}}{1989}]%
        {lecun1989optimal}
\bibfield{author}{\bibinfo{person}{Yann LeCun}, \bibinfo{person}{John~S.
  Denker}, {and} \bibinfo{person}{Sara~A. Solla}.}
  \bibinfo{year}{1989}\natexlab{}.
\newblock \showarticletitle{Optimal Brain Damage}. In
  \bibinfo{booktitle}{\emph{Advances in Neural Information Processing
  Systems}}.
\newblock


\bibitem[\protect\citeauthoryear{Lee, Ajanthan, and Torr}{Lee
  et~al\mbox{.}}{2019}]%
        {lee2019snip}
\bibfield{author}{\bibinfo{person}{Namhoon Lee},
  \bibinfo{person}{Thalaiyasingam Ajanthan}, {and} \bibinfo{person}{Philip
  H.~S. Torr}.} \bibinfo{year}{2019}\natexlab{}.
\newblock \showarticletitle{{SNIP}: Single-shot network pruning based on
  connection sensitivity}. In \bibinfo{booktitle}{\emph{International
  Conference on Learning Representations}}.
\newblock


\bibitem[\protect\citeauthoryear{Li, Kadav, Durdanovic, Samet, and Graf}{Li
  et~al\mbox{.}}{2017}]%
        {li2017pruning}
\bibfield{author}{\bibinfo{person}{Hao Li}, \bibinfo{person}{Asim Kadav},
  \bibinfo{person}{Igor Durdanovic}, \bibinfo{person}{Hanan Samet}, {and}
  \bibinfo{person}{Hans~Peter Graf}.} \bibinfo{year}{2017}\natexlab{}.
\newblock \showarticletitle{Pruning filters for efficient convnets}. In
  \bibinfo{booktitle}{\emph{International Conference on Learning
  Representations}}.
\newblock


\bibitem[\protect\citeauthoryear{Li and Talwalkar}{Li and Talwalkar}{2019}]%
        {li2019random}
\bibfield{author}{\bibinfo{person}{Liam Li} {and} \bibinfo{person}{Ameet
  Talwalkar}.} \bibinfo{year}{2019}\natexlab{}.
\newblock \showarticletitle{Random search and reproducibility for neural
  architecture search}.
\newblock \bibinfo{journal}{\emph{arXiv preprint arXiv:1902.07638}}
  (\bibinfo{year}{2019}).
\newblock


\bibitem[\protect\citeauthoryear{Liu, Simonyan, and Yang}{Liu
  et~al\mbox{.}}{2019}]%
        {liu2019darts}
\bibfield{author}{\bibinfo{person}{Hanxiao Liu}, \bibinfo{person}{Karen
  Simonyan}, {and} \bibinfo{person}{Yiming Yang}.}
  \bibinfo{year}{2019}\natexlab{}.
\newblock \showarticletitle{{DARTS}: Differentiable Architecture Search}. In
  \bibinfo{booktitle}{\emph{International Conference on Learning
  Representations}}.
\newblock


\bibitem[\protect\citeauthoryear{Luo, Tian, Qin, Chen, and Liu}{Luo
  et~al\mbox{.}}{2018}]%
        {luo2018neural}
\bibfield{author}{\bibinfo{person}{Renqian Luo}, \bibinfo{person}{Fei Tian},
  \bibinfo{person}{Tao Qin}, \bibinfo{person}{Enhong Chen}, {and}
  \bibinfo{person}{Tie-Yan Liu}.} \bibinfo{year}{2018}\natexlab{}.
\newblock \showarticletitle{Neural architecture optimization}. In
  \bibinfo{booktitle}{\emph{Advances in Neural Information Processing
  Systems}}.
\newblock


\bibitem[\protect\citeauthoryear{Mellor, Turner, Storkey, and Crowley}{Mellor
  et~al\mbox{.}}{2020}]%
        {mellor2020neural}
\bibfield{author}{\bibinfo{person}{Joseph Mellor}, \bibinfo{person}{Jack
  Turner}, \bibinfo{person}{Amos Storkey}, {and} \bibinfo{person}{Elliot~J
  Crowley}.} \bibinfo{year}{2020}\natexlab{}.
\newblock \showarticletitle{Neural Architecture Search without Training}.
\newblock \bibinfo{journal}{\emph{arXiv preprint arXiv:2006.04647}}
  (\bibinfo{year}{2020}).
\newblock


\bibitem[\protect\citeauthoryear{Misailovic, Roy, and Rinard}{Misailovic
  et~al\mbox{.}}{2011}]%
        {misailovic2011probabilistically}
\bibfield{author}{\bibinfo{person}{Sasa Misailovic}, \bibinfo{person}{Daniel~M
  Roy}, {and} \bibinfo{person}{Martin~C Rinard}.}
  \bibinfo{year}{2011}\natexlab{}.
\newblock \showarticletitle{Probabilistically accurate program
  transformations}. In \bibinfo{booktitle}{\emph{International Static Analysis
  Symposium}}. Springer, \bibinfo{pages}{316--333}.
\newblock


\bibitem[\protect\citeauthoryear{Mogers, Radu, Li, Turner, O'Boyle, and
  Dubach}{Mogers et~al\mbox{.}}{2020}]%
        {mogers2020automatic}
\bibfield{author}{\bibinfo{person}{Naums Mogers}, \bibinfo{person}{Valentin
  Radu}, \bibinfo{person}{Lu Li}, \bibinfo{person}{Jack Turner},
  \bibinfo{person}{Michael O'Boyle}, {and} \bibinfo{person}{Christophe
  Dubach}.} \bibinfo{year}{2020}\natexlab{}.
\newblock \showarticletitle{Automatic generation of specialized direct
  convolutions for mobile GPUs}. In \bibinfo{booktitle}{\emph{Proceedings of
  the 13th Annual Workshop on General Purpose Processing using Graphics
  Processing Unit}}. \bibinfo{pages}{41--50}.
\newblock


\bibitem[\protect\citeauthoryear{Molchanov, Mallya, Tyree, Frosio, and
  Kautz}{Molchanov et~al\mbox{.}}{2019}]%
        {molchanov2019importance}
\bibfield{author}{\bibinfo{person}{Pavlo Molchanov}, \bibinfo{person}{Arun
  Mallya}, \bibinfo{person}{Stephen Tyree}, \bibinfo{person}{Iuri Frosio},
  {and} \bibinfo{person}{Jan Kautz}.} \bibinfo{year}{2019}\natexlab{}.
\newblock \showarticletitle{Importance estimation for neural network pruning}.
  In \bibinfo{booktitle}{\emph{Proceedings of the IEEE Conference on Computer
  Vision and Pattern Recognition}}. \bibinfo{pages}{11264--11272}.
\newblock


\bibitem[\protect\citeauthoryear{Molchanov, Tyree, Karras, Aila, and
  Kautz}{Molchanov et~al\mbox{.}}{2017}]%
        {molchanov2017pruning}
\bibfield{author}{\bibinfo{person}{Pavlo Molchanov}, \bibinfo{person}{Stephen
  Tyree}, \bibinfo{person}{Tero Karras}, \bibinfo{person}{Timo Aila}, {and}
  \bibinfo{person}{Jan Kautz}.} \bibinfo{year}{2017}\natexlab{}.
\newblock \showarticletitle{Pruning Convolutional Neural Networks for Resource
  Efficient Inference}. In \bibinfo{booktitle}{\emph{International Conference
  on Learning Representations}}.
\newblock


\bibitem[\protect\citeauthoryear{Mullapudi, Vasista, and Bondhugula}{Mullapudi
  et~al\mbox{.}}{2015}]%
        {mullapudi2015polymage}
\bibfield{author}{\bibinfo{person}{Ravi~Teja Mullapudi}, \bibinfo{person}{Vinay
  Vasista}, {and} \bibinfo{person}{Uday Bondhugula}.}
  \bibinfo{year}{2015}\natexlab{}.
\newblock \showarticletitle{Polymage: Automatic optimization for image
  processing pipelines}. In \bibinfo{booktitle}{\emph{ACM SIGPLAN Notices}},
  Vol.~\bibinfo{volume}{50}. ACM, \bibinfo{pages}{429--443}.
\newblock


\bibitem[\protect\citeauthoryear{Park, Esmaeilzadeh, Zhang, Naik, and
  Harris}{Park et~al\mbox{.}}{2015}]%
        {park2015flexjava}
\bibfield{author}{\bibinfo{person}{Jongse Park}, \bibinfo{person}{Hadi
  Esmaeilzadeh}, \bibinfo{person}{Xin Zhang}, \bibinfo{person}{Mayur Naik},
  {and} \bibinfo{person}{William Harris}.} \bibinfo{year}{2015}\natexlab{}.
\newblock \showarticletitle{Flexjava: Language support for safe and modular
  approximate programming}. In \bibinfo{booktitle}{\emph{Proceedings of the
  2015 10th Joint Meeting on Foundations of Software Engineering}}.
  \bibinfo{pages}{745--757}.
\newblock


\bibitem[\protect\citeauthoryear{Peng, Xie, Zhang, Tan, and Wang}{Peng
  et~al\mbox{.}}{2018}]%
        {peng2018accelerating}
\bibfield{author}{\bibinfo{person}{Junran Peng}, \bibinfo{person}{Lingxi Xie},
  \bibinfo{person}{Zhaoxiang Zhang}, \bibinfo{person}{Tieniu Tan}, {and}
  \bibinfo{person}{Jingdong Wang}.} \bibinfo{year}{2018}\natexlab{}.
\newblock \showarticletitle{Accelerating Deep Neural Networks with Spatial
  Bottleneck Modules}.
\newblock \bibinfo{journal}{\emph{arXiv preprint arXiv:1809.02601}}
  (\bibinfo{year}{2018}).
\newblock


\bibitem[\protect\citeauthoryear{Pham, Guan, Zoph, Le, and Dean}{Pham
  et~al\mbox{.}}{2018}]%
        {pham2018efficient}
\bibfield{author}{\bibinfo{person}{Hieu Pham}, \bibinfo{person}{Melody~Y.
  Guan}, \bibinfo{person}{Barret Zoph}, \bibinfo{person}{Quoc~V. Le}, {and}
  \bibinfo{person}{Jeff Dean}.} \bibinfo{year}{2018}\natexlab{}.
\newblock \showarticletitle{Efficient neural architecture search via parameter
  sharing}. In \bibinfo{booktitle}{\emph{International Conference on Machine
  Learning}}.
\newblock


\bibitem[\protect\citeauthoryear{Pu, Bell, Yang, Setter, Richardson,
  Ragan-Kelley, and Horowitz}{Pu et~al\mbox{.}}{2017}]%
        {Pu:2017:PHS:3132652.3107953}
\bibfield{author}{\bibinfo{person}{Jing Pu}, \bibinfo{person}{Steven Bell},
  \bibinfo{person}{Xuan Yang}, \bibinfo{person}{Jeff Setter},
  \bibinfo{person}{Stephen Richardson}, \bibinfo{person}{Jonathan
  Ragan-Kelley}, {and} \bibinfo{person}{Mark Horowitz}.}
  \bibinfo{year}{2017}\natexlab{}.
\newblock \showarticletitle{Programming Heterogeneous Systems from an Image
  Processing DSL}.
\newblock \bibinfo{journal}{\emph{ACM Trans. Archit. Code Optim.}}
  \bibinfo{volume}{14}, \bibinfo{number}{3}, Article \bibinfo{articleno}{26}
  (\bibinfo{date}{Aug.} \bibinfo{year}{2017}), \bibinfo{numpages}{25}~pages.
\newblock
\showISSN{1544-3566}
\urldef\tempurl%
\url{https://doi.org/10.1145/3107953}
\showDOI{\tempurl}


\bibitem[\protect\citeauthoryear{Radu, Kaszyk, Wen, Turner, Cano, Crowley,
  Franke, Storkey, and O'Boyle}{Radu et~al\mbox{.}}{2019}]%
        {radu2019performance}
\bibfield{author}{\bibinfo{person}{Valentin Radu}, \bibinfo{person}{Kuba
  Kaszyk}, \bibinfo{person}{Yuan Wen}, \bibinfo{person}{Jack Turner},
  \bibinfo{person}{Jos{\'e} Cano}, \bibinfo{person}{Elliot~J Crowley},
  \bibinfo{person}{Bj{\"o}rn Franke}, \bibinfo{person}{Amos Storkey}, {and}
  \bibinfo{person}{Michael O'Boyle}.} \bibinfo{year}{2019}\natexlab{}.
\newblock \showarticletitle{Performance aware convolutional neural network
  channel pruning for embedded GPUs}. In \bibinfo{booktitle}{\emph{2019 IEEE
  International Symposium on Workload Characterization (IISWC)}}. IEEE,
  \bibinfo{pages}{24--34}.
\newblock


\bibitem[\protect\citeauthoryear{Ragan-Kelley, Barnes, Adams, Paris, Durand,
  and Amarasinghe}{Ragan-Kelley et~al\mbox{.}}{2013}]%
        {ragan2013halide}
\bibfield{author}{\bibinfo{person}{Jonathan Ragan-Kelley},
  \bibinfo{person}{Connelly Barnes}, \bibinfo{person}{Andrew Adams},
  \bibinfo{person}{Sylvain Paris}, \bibinfo{person}{Fr{\'e}do Durand}, {and}
  \bibinfo{person}{Saman Amarasinghe}.} \bibinfo{year}{2013}\natexlab{}.
\newblock \showarticletitle{Halide: a language and compiler for optimizing
  parallelism, locality, and recomputation in image processing pipelines}. In
  \bibinfo{booktitle}{\emph{Acm Sigplan Notices}}, Vol.~\bibinfo{volume}{48}.
  ACM, \bibinfo{pages}{519--530}.
\newblock


\bibitem[\protect\citeauthoryear{Rinard}{Rinard}{2006}]%
        {rinard2006probabilistic}
\bibfield{author}{\bibinfo{person}{Martin Rinard}.}
  \bibinfo{year}{2006}\natexlab{}.
\newblock \showarticletitle{Probabilistic accuracy bounds for fault-tolerant
  computations that discard tasks}. In \bibinfo{booktitle}{\emph{Proceedings of
  the 20th annual international conference on Supercomputing}}.
  \bibinfo{pages}{324--334}.
\newblock


\bibitem[\protect\citeauthoryear{Sampson, Baixo, Ransford, Moreau, Yip, Ceze,
  and Oskin}{Sampson et~al\mbox{.}}{2015}]%
        {sampson2015accept}
\bibfield{author}{\bibinfo{person}{Adrian Sampson}, \bibinfo{person}{Andr{\'e}
  Baixo}, \bibinfo{person}{Benjamin Ransford}, \bibinfo{person}{Thierry
  Moreau}, \bibinfo{person}{Joshua Yip}, \bibinfo{person}{Luis Ceze}, {and}
  \bibinfo{person}{Mark Oskin}.} \bibinfo{year}{2015}\natexlab{}.
\newblock \showarticletitle{Accept: A programmer-guided compiler framework for
  practical approximate computing}.
\newblock \bibinfo{journal}{\emph{University of Washington Technical Report
  UW-CSE-15-01}} \bibinfo{volume}{1}, \bibinfo{number}{2}
  (\bibinfo{year}{2015}).
\newblock


\bibitem[\protect\citeauthoryear{Schmidhuber}{Schmidhuber}{2015}]%
        {schmidhuber2015deep}
\bibfield{author}{\bibinfo{person}{J{\"u}rgen Schmidhuber}.}
  \bibinfo{year}{2015}\natexlab{}.
\newblock \showarticletitle{Deep learning in neural networks: An overview}.
\newblock \bibinfo{journal}{\emph{Neural networks}}  \bibinfo{volume}{61}
  (\bibinfo{year}{2015}), \bibinfo{pages}{85--117}.
\newblock


\bibitem[\protect\citeauthoryear{Sciuto, Yu, Jaggi, Musat, and Salzmann}{Sciuto
  et~al\mbox{.}}{2019}]%
        {sciuto2019evaluating}
\bibfield{author}{\bibinfo{person}{Christian Sciuto}, \bibinfo{person}{Kaicheng
  Yu}, \bibinfo{person}{Martin Jaggi}, \bibinfo{person}{Claudiu Musat}, {and}
  \bibinfo{person}{Mathieu Salzmann}.} \bibinfo{year}{2019}\natexlab{}.
\newblock \showarticletitle{Evaluating the Search Phase of Neural Architecture
  Search}.
\newblock \bibinfo{journal}{\emph{arXiv preprint arXiv:1902.08142}}
  (\bibinfo{year}{2019}).
\newblock


\bibitem[\protect\citeauthoryear{Sidiroglou-Douskos, Misailovic, Hoffmann, and
  Rinard}{Sidiroglou-Douskos et~al\mbox{.}}{2011}]%
        {sidiroglou2011managing}
\bibfield{author}{\bibinfo{person}{Stelios Sidiroglou-Douskos},
  \bibinfo{person}{Sasa Misailovic}, \bibinfo{person}{Henry Hoffmann}, {and}
  \bibinfo{person}{Martin Rinard}.} \bibinfo{year}{2011}\natexlab{}.
\newblock \showarticletitle{Managing performance vs. accuracy trade-offs with
  loop perforation}. In \bibinfo{booktitle}{\emph{Proceedings of the 19th ACM
  SIGSOFT symposium and the 13th European conference on Foundations of software
  engineering}}. \bibinfo{pages}{124--134}.
\newblock


\bibitem[\protect\citeauthoryear{So, Liang, and Le}{So et~al\mbox{.}}{2019}]%
        {so2019evolved}
\bibfield{author}{\bibinfo{person}{David~R So}, \bibinfo{person}{Chen Liang},
  {and} \bibinfo{person}{Quoc~V Le}.} \bibinfo{year}{2019}\natexlab{}.
\newblock \showarticletitle{The evolved transformer}.
\newblock \bibinfo{journal}{\emph{arXiv preprint arXiv:1901.11117}}
  (\bibinfo{year}{2019}).
\newblock


\bibitem[\protect\citeauthoryear{Steuwer, Remmelg, and Dubach}{Steuwer
  et~al\mbox{.}}{2017}]%
        {steuwer2017lift}
\bibfield{author}{\bibinfo{person}{Michel Steuwer}, \bibinfo{person}{Toomas
  Remmelg}, {and} \bibinfo{person}{Christophe Dubach}.}
  \bibinfo{year}{2017}\natexlab{}.
\newblock \showarticletitle{Lift: a functional data-parallel IR for
  high-performance GPU code generation}. In
  \bibinfo{booktitle}{\emph{International Symposium on Code Generation and
  Optimization}}.
\newblock


\bibitem[\protect\citeauthoryear{Tan, Chen, Pang, Vasudevan, Sandler, Howard,
  and Le}{Tan et~al\mbox{.}}{2019}]%
        {tan2019mnasnet}
\bibfield{author}{\bibinfo{person}{Mingxing Tan}, \bibinfo{person}{Bo Chen},
  \bibinfo{person}{Ruoming Pang}, \bibinfo{person}{Vijay Vasudevan},
  \bibinfo{person}{Mark Sandler}, \bibinfo{person}{Andrew Howard}, {and}
  \bibinfo{person}{Quoc~V. Le}.} \bibinfo{year}{2019}\natexlab{}.
\newblock \showarticletitle{Mnas{N}et: Platform-aware neural architecture
  search for mobile}. In \bibinfo{booktitle}{\emph{Proceedings of the IEEE
  Conference on Computer Vision and Pattern Recognition}}.
\newblock


\bibitem[\protect\citeauthoryear{Tan and Le}{Tan and Le}{2019}]%
        {tan2019efficientnet}
\bibfield{author}{\bibinfo{person}{Mingxing Tan} {and} \bibinfo{person}{Quoc~V.
  Le}.} \bibinfo{year}{2019}\natexlab{}.
\newblock \showarticletitle{EfficientNet: Rethinking Model Scaling for
  Convolutional Neural Networks}. In \bibinfo{booktitle}{\emph{International
  Conference on Machine Learning}}.
\newblock


\bibitem[\protect\citeauthoryear{Theis, Korshunova, Tejani, and
  Husz{\'a}r}{Theis et~al\mbox{.}}{2018}]%
        {theis2018faster}
\bibfield{author}{\bibinfo{person}{Lucas Theis}, \bibinfo{person}{Iryna
  Korshunova}, \bibinfo{person}{Alykhan Tejani}, {and} \bibinfo{person}{Ferenc
  Husz{\'a}r}.} \bibinfo{year}{2018}\natexlab{}.
\newblock \bibinfo{title}{Faster gaze prediction with dense networks and
  {F}isher pruning}.
\newblock
\newblock


\bibitem[\protect\citeauthoryear{Tillet, Kung, and Cox}{Tillet
  et~al\mbox{.}}{2019}]%
        {tillet2019triton}
\bibfield{author}{\bibinfo{person}{Philippe Tillet}, \bibinfo{person}{HT Kung},
  {and} \bibinfo{person}{David Cox}.} \bibinfo{year}{2019}\natexlab{}.
\newblock \showarticletitle{Triton: an intermediate language and compiler for
  tiled neural network computations}. In
  \bibinfo{booktitle}{\emph{International Workshop on Machine Learning and
  Programming Languages}}.
\newblock


\bibitem[\protect\citeauthoryear{Turner, Crowley, O'Boyle, Storkey, and
  Gray}{Turner et~al\mbox{.}}{2020}]%
        {Turner2020BlockSwap:}
\bibfield{author}{\bibinfo{person}{Jack Turner}, \bibinfo{person}{Elliot~J.
  Crowley}, \bibinfo{person}{Michael O'Boyle}, \bibinfo{person}{Amos Storkey},
  {and} \bibinfo{person}{Gavin Gray}.} \bibinfo{year}{2020}\natexlab{}.
\newblock \showarticletitle{BlockSwap: Fisher-guided Block Substitution for
  Network Compression on a Budget}. In \bibinfo{booktitle}{\emph{International
  Conference on Learning Representations}}.
\newblock
\urldef\tempurl%
\url{https://openreview.net/forum?id=SklkDkSFPB}
\showURL{%
\tempurl}


\bibitem[\protect\citeauthoryear{Turner, Crowley, Radu, Cano, Storkey, and
  O'Boyle}{Turner et~al\mbox{.}}{2018}]%
        {turner2018distilling}
\bibfield{author}{\bibinfo{person}{Jack Turner}, \bibinfo{person}{Elliot~J
  Crowley}, \bibinfo{person}{Valentin Radu}, \bibinfo{person}{Jos{\'e} Cano},
  \bibinfo{person}{Amos Storkey}, {and} \bibinfo{person}{Michael O'Boyle}.}
  \bibinfo{year}{2018}\natexlab{}.
\newblock \showarticletitle{Distilling with performance enhanced students}.
\newblock \bibinfo{journal}{\emph{arXiv preprint arXiv:1810.10460}}
  (\bibinfo{year}{2018}).
\newblock


\bibitem[\protect\citeauthoryear{Vasilache, Bastoul, and Cohen}{Vasilache
  et~al\mbox{.}}{2006}]%
        {vasilache2006polyhedral}
\bibfield{author}{\bibinfo{person}{Nicolas Vasilache},
  \bibinfo{person}{C{\'e}dric Bastoul}, {and} \bibinfo{person}{Albert Cohen}.}
  \bibinfo{year}{2006}\natexlab{}.
\newblock \showarticletitle{Polyhedral code generation in the real world}. In
  \bibinfo{booktitle}{\emph{International Conference on Compiler
  Construction}}.
\newblock


\bibitem[\protect\citeauthoryear{Vasilache, Zinenko, Theodoridis, Goyal,
  DeVito, Moses, Verdoolaege, Adams, and Cohen}{Vasilache
  et~al\mbox{.}}{2018}]%
        {vasilache2018tensor}
\bibfield{author}{\bibinfo{person}{Nicolas Vasilache},
  \bibinfo{person}{Oleksandr Zinenko}, \bibinfo{person}{Theodoros Theodoridis},
  \bibinfo{person}{Priya Goyal}, \bibinfo{person}{Zachary DeVito},
  \bibinfo{person}{William~S. Moses}, \bibinfo{person}{Sven Verdoolaege},
  \bibinfo{person}{Andrew Adams}, {and} \bibinfo{person}{Albert Cohen}.}
  \bibinfo{year}{2018}\natexlab{}.
\newblock \showarticletitle{Tensor Comprehensions: Framework-Agnostic
  High-Performance Machine Learning Abstractions}.
\newblock \bibinfo{journal}{\emph{arXiv preprint arXiv:1802.04730}}
  (\bibinfo{year}{2018}).
\newblock


\bibitem[\protect\citeauthoryear{Verdoolaege}{Verdoolaege}{2010}]%
        {verdoolaege2010isl}
\bibfield{author}{\bibinfo{person}{Sven Verdoolaege}.}
  \bibinfo{year}{2010}\natexlab{}.
\newblock \showarticletitle{isl: An integer set library for the polyhedral
  model}. In \bibinfo{booktitle}{\emph{International Congress on Mathematical
  Software}}.
\newblock


\bibitem[\protect\citeauthoryear{Verdoolaege, Carlos~Juega, Cohen,
  Ignacio~Gomez, Tenllado, and Catthoor}{Verdoolaege et~al\mbox{.}}{2013}]%
        {verdoolaege2013polyhedral}
\bibfield{author}{\bibinfo{person}{Sven Verdoolaege}, \bibinfo{person}{Juan
  Carlos~Juega}, \bibinfo{person}{Albert Cohen}, \bibinfo{person}{Jose
  Ignacio~Gomez}, \bibinfo{person}{Christian Tenllado}, {and}
  \bibinfo{person}{Francky Catthoor}.} \bibinfo{year}{2013}\natexlab{}.
\newblock \showarticletitle{Polyhedral parallel code generation for CUDA}.
\newblock \bibinfo{journal}{\emph{ACM Transactions on Architecture and Code
  Optimization (TACO)}} \bibinfo{volume}{9}, \bibinfo{number}{4}
  (\bibinfo{year}{2013}), \bibinfo{pages}{54}.
\newblock


\bibitem[\protect\citeauthoryear{Verdoolaege, Guelton, Grosser, and
  Cohen}{Verdoolaege et~al\mbox{.}}{2014}]%
        {verdoolaege2014schedule}
\bibfield{author}{\bibinfo{person}{Sven Verdoolaege}, \bibinfo{person}{Serge
  Guelton}, \bibinfo{person}{Tobias Grosser}, {and} \bibinfo{person}{Albert
  Cohen}.} \bibinfo{year}{2014}\natexlab{}.
\newblock \showarticletitle{Schedule trees}. In
  \bibinfo{booktitle}{\emph{International Workshop on Polyhedral Compilation
  Techniques}}.
\newblock


\bibitem[\protect\citeauthoryear{Wistuba, Rawat, and Pedapati}{Wistuba
  et~al\mbox{.}}{2019}]%
        {wistuba2019survey}
\bibfield{author}{\bibinfo{person}{Martin Wistuba}, \bibinfo{person}{Ambrish
  Rawat}, {and} \bibinfo{person}{Tejaswini Pedapati}.}
  \bibinfo{year}{2019}\natexlab{}.
\newblock \showarticletitle{A survey on neural architecture search}.
\newblock \bibinfo{journal}{\emph{arXiv preprint arXiv:1905.01392}}
  (\bibinfo{year}{2019}).
\newblock


\bibitem[\protect\citeauthoryear{Wu, Dai, Zhang, Wang, Sun, Wu, Tian, Vajda,
  Jia, and Keutzer}{Wu et~al\mbox{.}}{2019}]%
        {wu2019fbnet}
\bibfield{author}{\bibinfo{person}{Bichen Wu}, \bibinfo{person}{Xiaoliang Dai},
  \bibinfo{person}{Peizhao Zhang}, \bibinfo{person}{Yanghan Wang},
  \bibinfo{person}{Fei Sun}, \bibinfo{person}{Yiming Wu},
  \bibinfo{person}{Yuandong Tian}, \bibinfo{person}{Peter Vajda},
  \bibinfo{person}{Yangqing Jia}, {and} \bibinfo{person}{Kurt Keutzer}.}
  \bibinfo{year}{2019}\natexlab{}.
\newblock \showarticletitle{{FBN}et: Hardware-aware efficient convnet design
  via differentiable neural architecture search}. In
  \bibinfo{booktitle}{\emph{Proceedings of the IEEE Conference on Computer
  Vision and Pattern Recognition}}.
\newblock


\bibitem[\protect\citeauthoryear{Xie, Girshick, Doll{\'a}r, Tu, and He}{Xie
  et~al\mbox{.}}{2017}]%
        {xie2017aggregated}
\bibfield{author}{\bibinfo{person}{Saining Xie}, \bibinfo{person}{Ross
  Girshick}, \bibinfo{person}{Piotr Doll{\'a}r}, \bibinfo{person}{Zhuowen Tu},
  {and} \bibinfo{person}{Kaiming He}.} \bibinfo{year}{2017}\natexlab{}.
\newblock \showarticletitle{Aggregated Residual Transformations for Deep Neural
  Networks}. In \bibinfo{booktitle}{\emph{Proceedings of the IEEE Conference on
  Computer Vision and Pattern Recognition}}.
\newblock


\bibitem[\protect\citeauthoryear{Yang, Howard, Chen, Zhang, Go, Sandler, Sze,
  and Adam}{Yang et~al\mbox{.}}{2018}]%
        {yang2018netadapt}
\bibfield{author}{\bibinfo{person}{Tien-Ju Yang}, \bibinfo{person}{Andrew
  Howard}, \bibinfo{person}{Bo Chen}, \bibinfo{person}{Xiao Zhang},
  \bibinfo{person}{Alec Go}, \bibinfo{person}{Mark Sandler},
  \bibinfo{person}{Vivienne Sze}, {and} \bibinfo{person}{Hartwig Adam}.}
  \bibinfo{year}{2018}\natexlab{}.
\newblock \showarticletitle{Netadapt: Platform-aware neural network adaptation
  for mobile applications}. In \bibinfo{booktitle}{\emph{Proceedings of the
  European Conference on Computer Vision (ECCV)}}.
\newblock


\bibitem[\protect\citeauthoryear{Yang, Pu, Rister, Bhagdikar, Richardson,
  Kvatinsky, Ragan-Kelley, Pedram, and Horowitz}{Yang et~al\mbox{.}}{2016}]%
        {yang2016systematic}
\bibfield{author}{\bibinfo{person}{Xuan Yang}, \bibinfo{person}{Jing Pu},
  \bibinfo{person}{Blaine~Burton Rister}, \bibinfo{person}{Nikhil Bhagdikar},
  \bibinfo{person}{Stephen Richardson}, \bibinfo{person}{Shahar Kvatinsky},
  \bibinfo{person}{Jonathan Ragan-Kelley}, \bibinfo{person}{Ardavan Pedram},
  {and} \bibinfo{person}{Mark Horowitz}.} \bibinfo{year}{2016}\natexlab{}.
\newblock \showarticletitle{A Systematic Approach to Blocking Convolutional
  Neural Networks}.
\newblock \bibinfo{journal}{\emph{arXiv preprint arXiv:1606.04209}}
  (\bibinfo{year}{2016}).
\newblock


\bibitem[\protect\citeauthoryear{Ying, Klein, Real, Christiansen, Murphy, and
  Hutter}{Ying et~al\mbox{.}}{2019}]%
        {ying2019bench}
\bibfield{author}{\bibinfo{person}{Chris Ying}, \bibinfo{person}{Aaron Klein},
  \bibinfo{person}{Esteban Real}, \bibinfo{person}{Eric Christiansen},
  \bibinfo{person}{Kevin Murphy}, {and} \bibinfo{person}{Frank Hutter}.}
  \bibinfo{year}{2019}\natexlab{}.
\newblock \showarticletitle{Nas-bench-101: Towards reproducible neural
  architecture search}.
\newblock \bibinfo{journal}{\emph{arXiv preprint arXiv:1902.09635}}
  (\bibinfo{year}{2019}).
\newblock


\bibitem[\protect\citeauthoryear{Zela, Siems, and Hutter}{Zela
  et~al\mbox{.}}{2020}]%
        {zela2020bench}
\bibfield{author}{\bibinfo{person}{Arber Zela}, \bibinfo{person}{Julien Siems},
  {and} \bibinfo{person}{Frank Hutter}.} \bibinfo{year}{2020}\natexlab{}.
\newblock \showarticletitle{NAS-Bench-1Shot1: Benchmarking and Dissecting
  One-shot Neural Architecture Search}.
\newblock \bibinfo{journal}{\emph{arXiv preprint arXiv:2001.10422}}
  (\bibinfo{year}{2020}).
\newblock


\bibitem[\protect\citeauthoryear{Zerrell and Bruestle}{Zerrell and
  Bruestle}{2019}]%
        {zerrell2019stripe}
\bibfield{author}{\bibinfo{person}{Tim Zerrell} {and} \bibinfo{person}{Jeremy
  Bruestle}.} \bibinfo{year}{2019}\natexlab{}.
\newblock \showarticletitle{Stripe: Tensor Compilation via the Nested
  Polyhedral Model}.
\newblock \bibinfo{journal}{\emph{arXiv preprint arXiv:1903.06498}}
  (\bibinfo{year}{2019}).
\newblock


\bibitem[\protect\citeauthoryear{Zheng, Jia, Sun, Wu, Yu, Haj-Ali, Wang, Yang,
  Zhuo, Sen, et~al\mbox{.}}{Zheng et~al\mbox{.}}{2020}]%
        {zheng2020ansor}
\bibfield{author}{\bibinfo{person}{Lianmin Zheng}, \bibinfo{person}{Chengfan
  Jia}, \bibinfo{person}{Minmin Sun}, \bibinfo{person}{Zhao Wu},
  \bibinfo{person}{Cody~Hao Yu}, \bibinfo{person}{Ameer Haj-Ali},
  \bibinfo{person}{Yida Wang}, \bibinfo{person}{Jun Yang},
  \bibinfo{person}{Danyang Zhuo}, \bibinfo{person}{Koushik Sen},
  {et~al\mbox{.}}} \bibinfo{year}{2020}\natexlab{}.
\newblock \showarticletitle{Ansor: Generating High-Performance Tensor Programs
  for Deep Learning}.
\newblock \bibinfo{journal}{\emph{arXiv preprint arXiv:2006.06762}}
  (\bibinfo{year}{2020}).
\newblock


\bibitem[\protect\citeauthoryear{Zhu, Misailovic, Kelner, and Rinard}{Zhu
  et~al\mbox{.}}{2012}]%
        {zhu2012randomized}
\bibfield{author}{\bibinfo{person}{Zeyuan~Allen Zhu}, \bibinfo{person}{Sasa
  Misailovic}, \bibinfo{person}{Jonathan~A Kelner}, {and}
  \bibinfo{person}{Martin Rinard}.} \bibinfo{year}{2012}\natexlab{}.
\newblock \showarticletitle{Randomized accuracy-aware program transformations
  for efficient approximate computations}.
\newblock \bibinfo{journal}{\emph{ACM SIGPLAN Notices}} \bibinfo{volume}{47},
  \bibinfo{number}{1} (\bibinfo{year}{2012}), \bibinfo{pages}{441--454}.
\newblock


\bibitem[\protect\citeauthoryear{Zoph and Le}{Zoph and Le}{2017}]%
        {zoph2017neural}
\bibfield{author}{\bibinfo{person}{Barret Zoph} {and} \bibinfo{person}{Quoc~V.
  Le}.} \bibinfo{year}{2017}\natexlab{}.
\newblock \showarticletitle{Neural architecture search with reinforcement
  learning}. In \bibinfo{booktitle}{\emph{International Conference on Learning
  Representations}}.
\newblock


\end{thebibliography}
