CrossStack: A 3-D Reconfigurable RRAM Crossbar Inference Engine
Jason~K.~Eshraghian$^1$
~\IEEEmembership{Member
~IEEE
Deep neural network inference accelerators are rapidly growing in importance as we turn to massively parallelized processing beyond GPUs and ASICs. The dominant operation in feedforward inference is the multiply-and-accumlate process, where each column in a crossbar generates the current response of a single neuron. As a result, memristor crossbar arrays parallelize inference and image processing tasks very efficiently. In this brief, we present a 3-D active memristor crossbar array `CrossStack', which adopts stacked pairs of \ce{Al/TiO2/\ce{TiO_{2-x/\ce{Al devices with common middle electrodes. By designing CMOS-memristor hybrid cells used in the layout of the array, CrossStack can operate in one of two user-configurable modes as a reconfigurable inference engine: 1) expansion mode and 2) deep-net mode. In expansion mode, the resolution of the network is doubled by increasing the number of inputs for a given chip area, reducing IR drop by 22\%. In deep-net mode, inference speed per-10-bit convolution is improved by 29\% by simultaneously using one \ce{TiO2/\ce{TiO_{2-x layer for read processes, and the other for write processes. We experimentally verify both modes on our $10\times10\times2$ array.
bare_jrnl_arxiv.tex
\documentclass[conference]{IEEEtran}
\ifCLASSINFOpdf
\else
\fi

\ifCLASSOPTIONcompsoc
    \usepackage[caption=false, font=normalsize, labelfont=sf, textfont=sf]{subfig}
\else
	\usepackage[caption=false, font=footnotesize]{subfig}
\fi

\hyphenation{op-tical net-works semi-conduc-tor}
\usepackage{cite}
\usepackage[version=4]{mhchem}
\usepackage{graphicx}
\usepackage{amsmath,amssymb,amsfonts}
\usepackage{algorithm}
\usepackage[noend]{algpseudocode}
\usepackage{tablefootnote}
\bibliographystyle{IEEEtran}
\usepackage[table,xcdraw]{xcolor}
\usepackage{lipsum}
\usepackage{textcomp}
\usepackage{tikz}
\usepackage[utf8]{inputenc}
\usepackage{pgfplots} 
\usepackage{pgfgantt}
\usepackage{pdflscape}
\pgfplotsset{compat=newest} 
\pgfplotsset{plot coordinates/math parser=false}
\usetikzlibrary{plotmarks}

%\makeatletter
%\def\BState{\State\hskip-\ALG@thistlm}
%\makeatother

\begin{document}
\title{CrossStack: A 3-D Reconfigurable RRAM Crossbar Inference Engine}

\author{Jason~K.~Eshraghian$^1$,~\IEEEmembership{Member,~IEEE,}
		Kyoungrok~Cho$^2$~\IEEEmembership{Senior~Member,~IEEE,}
        and~Sung~Mo~Kang$^3$~\IEEEmembership{Life~Fellow,~IEEE,}\\%
        
        \IEEEauthorblockA{$^1$\textit{School of Electrical, Electronic and Computer Engineering, University of Michigan, Ann Arbor, MI 48109 USA}}
        \IEEEauthorblockA{$^2$\textit{College of Electrical and Computer Engineering, Chungbuk National University, Cheongju 362763, South Korea}}
\IEEEauthorblockA{$^3$\textit{Jack Baskin School of Engineering, University of California, Santa Cruz, Santa Cruz, CA 95064 USA}}}%

% The paper headers
%\markboth{IEEE Transactions on Circuits and systems-II: Express Briefs, Vol. XX, No. Y, Month Year}%
%{Eshraghian \MakeLowercase{\textit{et al.}}: CrossStack: A 2.5-D Reconfigurable RRAM Crossbar Inference Engine}

\maketitle

\begin{abstract}
Deep neural network inference accelerators are rapidly growing in importance as we turn to massively parallelized processing beyond GPUs and ASICs. The dominant operation in feedforward inference is the multiply-and-accumlate process, where each column in a crossbar generates the current response of a single neuron. As a result, memristor crossbar arrays parallelize inference and image processing tasks very efficiently. In this brief, we present a 3-D active memristor crossbar array `CrossStack', which adopts stacked pairs of \ce{Al/TiO2}/\ce{TiO_{2-x}}/\ce{Al} devices with common middle electrodes. By designing CMOS-memristor hybrid cells used in the layout of the array, CrossStack can operate in one of two user-configurable modes as a reconfigurable inference engine: 1) expansion mode and 2) deep-net mode. In expansion mode, the resolution of the network is doubled by increasing the number of inputs for a given chip area, reducing IR drop by 22\%. In deep-net mode, inference speed per-10-bit convolution is improved by 29\% by simultaneously using one \ce{TiO2}/\ce{TiO_{2-x}} layer for read processes, and the other for write processes. We experimentally verify both modes on our $10\times10\times2$ array.
\end{abstract}

% Note that keywords are not normally used for peerreview papers.
\begin{IEEEkeywords}
deep learning, in-memory computing, memristors, neural network, RRAM.
\end{IEEEkeywords}

%\IEEEpeerreviewmaketitle

\section{Introduction}
\IEEEPARstart{I}{ncreasing} the sizes of artificial neural networks (ANNs) has been the most common response to the copious amount of data being continuously generated. Where training sets are in excess of billions of inputs processed through hundreds of millions of parameters in a neural network \cite{Mahajan2018}, new ways to speed up the processing of all this information must be developed. Since 2012, the training runtime of neural networks has doubled every 3--4 months. It is equally important to develop hardware that is not only optimized for running very large scale networks, but is adaptable to the unceasing wave of emerging ANN topologies.

Memristors are now ubiquitous in neuromorphic computing literature due to their long retention \cite{Kim2010, Kumar2017, Lin20201}, excellent scalability \cite{Pi2019, Fuller2019}, fast read and write speeds \cite{Merced2016, Lin20202}, compatibility with CMOS technology \cite{Eshraghian2018, Chakrabarti2017, Cai2019, Azghadi2020}, and precise weight updates \cite{Lammie2020, Krestinskaya2018}. The development of dense integrated structures with 3-D stacked crossbar arrays enables an increase in the throughput for a given chip area, but thus far, target applications of 3-D RRAM have mostly been limited to digital memory \cite{Chevallier2010, Wang2011, Jo2014, Hong2018, Azghadi20201, Baek2019, Fastow2019, Eshraghian2017}.

It can be difficult for ASIC designs to keep pace with the latest developments in machine learning due to the lag time between algorithm development and the full IC design cycle. With popular machine learning methods in a rapidly evolving state, reconfigurability is of paramount importance to ensure hardware does not become obsolete the moment new network architectures and topologies are introduced. In response to this, we present a reconfigurable stacked pair of memristor crossbars dubbed `CrossStack' that can be operated in one of two modes: 1) expansion mode, and 2) deep-net mode. In expansion mode, the resolution of the network is doubled by increasing the number of inputs for a given chip area, thus reducing IR drop by 22\% of an equivalent array. In deep-net mode, inference speed per-10-bit convolution is improved by 29\% by simultaneously using one array for read processes, and the other for write processes. This brief will demonstrate how to selectively isolate and couple the two layers using CMOS cell design. We experimentally verify this on our in-house fabricated crossbar stack, using separately controlled CMOS circuitry in the SK Hynix 180nm process. 

\section{Matrix-Vector Multiplication}
To perform analog-domain multiply-and-accumulate (MAC) using RRAM arrays, %inputs are multiplied by weights and then summed together in the neuron. %The output passes the summation of [inputs $\times$ weights] into an activation function before being sent to the neurons in the next hidden layer. 
a voltage vector $\boldsymbol V^i$ is applied at the input, multiplied by a conductance matrix $\boldsymbol G$, to generate a current vector of $\boldsymbol i=\boldsymbol V^i \boldsymbol G$ in accordance with Ohm's Law and Kirchhoff's Current Law. On a pre-trained network, the conductance of each memristor is programmed prior to read-out. For further detail on MAC on a crossbar, we recommend referring to \cite{Hu2016,Shafiee2016}. In a conventional crossbar, the memristors must be programmed prior to read-out. Where the number of parameters exceed the memory resources available, RRAM cells must be reprogrammed while the data flow of the activations is stalled. 
CrossStack avoids possible stalling that may occur by adding the option to pipeline the read and write processes simultaneously across the two layers. How this is achieved will be described in the following section.


\section{Circuit Operation Modes}


\begin{figure}
\centering
\subfloat[]
{
	\includegraphics[scale=.7]{Fig2a}
	\label{fig:2a}
}
\\
\subfloat[]
{
	\includegraphics[scale=.38]{Fig2b}
	\label{fig:2b}
}
\subfloat[]
{
	\includegraphics[scale=.38]{Fig2c}
	\label{fig:2c}
}
\subfloat[]
{
	\includegraphics[scale=.38]{Fig2d}
	\label{fig:2d}
}\\
\subfloat[]
{
	\includegraphics[scale=.55]{Fig2e}
	\label{fig:2e}
}\\
\subfloat[]
{
	\includegraphics[scale=.55]{Fig2f}
	\label{fig:2f}
}
\caption{(a) Simplified 3-D memristor crossbar array in expansion mode with cumulative current through the shared electrode (b) Memristor-CMOS cell (c) Current flow in read mode when read-enable is set high (d) Current flow in write mode when read-enable is set low (e) A stacked pair of cells during the read cycle in expansion mode (f) A stacked pair of cells in deep-net mode.}
\label{fig:2}
\end{figure}

A simplified structure of CrossStack is depicted in Fig.~1(a), and the memristor-CMOS cell schematic is given in Fig.~1(b). This work presents two modes in which CrossStack may operate in. These modes are controlled by an active-high read-enable signal \texttt{RE}. 

 
\subsection{Expansion Mode} 
Expansion mode enables access to a shared column line from memristors both above and below the wire. Vertical stacking of memristors doubles the number of possible inputs and weights to each neuron for a given length of column wire which is illustrated in Fig. 1(a) when compared to conventional crossbars. This can be formalized by the following equation:

\begin{equation}
\label{Eq1}
\begin{bmatrix} 
i_1 & i_2 & ... & i_m
\end{bmatrix} = 
\begin{bmatrix} 
V_1^i \\ V_2^i \\ \vdots \\ V_n^i 
\end{bmatrix}^T 
\begin{bmatrix} 
G_{1,1} & G_{1,2} & ... & G_{1,m} \\
G_{2,1} & G_{2,2} & ... & G_{2,m} \\ 
\vdots & \vdots & \ddots & \vdots \\ 
G_{n,1} & G_{n,2} & ... & G_{n,m}
\end{bmatrix}
\end{equation}

\noindent where $m$ is the number of columns in the crossbar and $n$ is the number of rows. In expansion mode, $n$ is double that of a 2-D array, as there are rows both above and below the shared column contributing to output current.

%The wire resistance between each cell was calculated to be Rwire = 2.82 Ω, which is negligible in our small-scale crossbar of 10 cells-per-layer. We measured IR drop at the critical case by applying a programming voltage of VIN = 1.8V, and switching all memristors on, where RS = 1 kΩ and found the drop across 10 column wires (20 stacked MRs) to be 0.173 V. An equivalent, 2-D crossbar across 20 column wires (20 planar MRs) was simulated in Cadence Virtuoso to have a drop of 0.57 V, which makes our 2.5-D chip 22\% less likely to suffer from write failures.

%\begin{figure}
%\includegraphics[scale=.45]{Fig3.png}
%\caption{Test}
%\end{figure}

To activate a cell in expansion mode, the read-enable signal of all cells must be identical. The current pathway from input to output of a single cell is depicted in Fig.~1(c), and a pair of stacked cells is shown in Fig.~1(e). To generate a read-out current at each shared column, \texttt{RE} must be set high (in our case, $V \geq V_{Th} = 0.4 V$; described in further detail in our experimental results). If transistors \texttt{N1} and \texttt{N2} are treated as switches, then \texttt{N2} would be off and \texttt{N1} would be on. Therefore, a current pathway is formed from both upper and lower crossbar arrays to the column line. To program a memristor, \texttt{RE} is set low as in Fig.~1(d). This causes \texttt{N2} to switch on and \texttt{N1} off, which forms a pathway from the memristor to ground and prevents current from flowing to the shared column. Therefore, the two crossbars can be programmed independently of one another by isolating them. The transistors are sized to ensure a negligible leakage current, and to sustain a sufficiently low \texttt{ON} resistance in comparison to the memristance while it is operating in the linear region such that it behaves as an ideal switch.


\subsection{Deep-net Mode}

% In deep-net mode on CrossStack, we have access to two stacked 10 × 10 arays similar to that shown in Fig. 3(c) (without CMOS of the MR-CMOS cell shown for simplicity). One layer is programmed in anticipation of the current read-out being generated from the other layer, enabling a heightened level of parallelism in MAC processing.

Deep-net mode ensures both pairs of arrays are isolated from one another at all times. Isolation biasing enables each layer to operate independently. The two arrays must have complementary \texttt{RE} signals as distinct from expansion mode, depicted in Fig.~1(f). This means that while one array generates a read-out current, the other array of memristors are being programmed (in write mode). As described above, when \texttt{RE} is low the cell does not contribute to read-out current and input voltage $V^{i}$ for the write layer is applied such that the conductances written to the memristors correspond to the weights of the next hidden layer in the neural network. Once the analog output current is digitized as a voltage, the write-layer has been pre-programmed and there is no need to buffer the current or store it in memory, as is required by most other pipelines \cite{Shafiee2016, Song2017, Valavi2019}. This process is repeated, but now the roles of the crossbars are reversed. The original read-layer is now programmed to the next hidden layer of weights, and the original write-layer generates the read-out current. Thus, read-write processes run in parallel. One layer is programmed in anticipation of the output from the other layer, which enables a novel in-situ pipeline.

In describing how to program a memristor, the key difference between expansion and deep-net modes is that in expansion, all cells are identically biased for either read or write at any given time. In deep-net mode, 50\% of the cells are biased for read processes and the remaining 50\% are biased for write processes, only switching once each operation is complete. The shorter read-out time is subsumed within the programming cycle, but at the expense of half the number of inputs $n$ in (1). We quantitatively demonstrate this in our experimental results.

\section{Experimental Results}


\subsection{Crossbar Fabrication}

\begin{figure}
\includegraphics[scale=1]{Fig3.png}
\caption{A $10 \times 10 \times 2$ prototype of CrossStack, with a cross-sectional view of the active layer taken using a focus ion beam analyzer.}
\end{figure}
CrossStack was constructed based on two monolithically integrated crossbar arrays with a shared central electrode which make up the column line, based on a sandwich structure of \ce{Al}/\ce{TiO2}/\ce{TiO_{2-x}}/\ce{Al} layers. A layer of Al (200-nm-thick and 20-$\mu$m-wide) was deposited using photolithography on a glass wafer as the bottom electrode (irradiated using mask alignment for 100~s, subsequently developed at  23$^{\circ}$C for 120~s). Any excess Al outside of the channel region was removed via wet etching (\ce{H3PO4} : \ce{HNO3} : \ce{CH3COOH} : \ce{H2O} = 80 ml : 5 ml : 5 ml : 10 ml) at a rate of $\Delta$d/t = 300 nm/min. \ce{TiO2} (5-nm-thick) and \ce{TiO_{2-x}} (15-nm-thick) thin films were formed by atomic layer deposition and magnetron sputtering to fabricate the memristor. Another 200-nm-thick layer of Al was sputtered as the top electrode using photolithography to create a 20~$\mu$m $\times$ 20~$\mu$m mask. After a planarization step, the top stack of active layer and metal were also deposited. Note that the polarity of the pair of active layers are mirrored, as distinguishable from \cite{Adam2017}. This allows for identical input voltages to be applied when programming the memristors. Fig.~2 shows a working $10 \times 10 \times 2$ prototype of CrossStack, and a cross-sectional view of the memristor taken using a focus ion beam analyzer.

\begin{table}[]\scriptsize
\centering\caption{CrossStack Characteristics}
\begin{tabular}{
>{\columncolor[HTML]{FFFFFF}}l 
>{\columncolor[HTML]{FFFFFF}}l 
>{\columncolor[HTML]{FFFFFF}}l }
\hline\vspace{-6pt}
 &  &  \\ \hline
\multicolumn{1}{|c|}{\cellcolor[HTML]{FFFFFF}Symbol} & \multicolumn{1}{l}{\cellcolor[HTML]{FFFFFF}Parameter} & \multicolumn{1}{r|}{\cellcolor[HTML]{FFFFFF}Value} \\ \hline
\multicolumn{1}{|c|}{\cellcolor[HTML]{FFFFFF}$R_s$} & \multicolumn{1}{l}{\cellcolor[HTML]{FFFFFF}static resistance of set; \color{red}\textit{$R_s=\frac{1}{g_s}$}} & \multicolumn{1}{r|}{\cellcolor[HTML]{FFFFFF}10K$\Omega~\pm$ 7\%} \\
\multicolumn{1}{|c|}{\cellcolor[HTML]{FFFFFF}$R_{r}$} & \multicolumn{1}{l}{\cellcolor[HTML]{FFFFFF}static resistance of reset;\color{red}\textit{$R_r=\frac{1}{g_r}$}} & \multicolumn{1}{r|}{\cellcolor[HTML]{FFFFFF}100K$\Omega~\pm$ 10\%} \\
\multicolumn{1}{|c|}{\cellcolor[HTML]{FFFFFF}$V_{DD}$} & \multicolumn{1}{l}{\cellcolor[HTML]{FFFFFF}supply voltage} & \multicolumn{1}{r|}{\cellcolor[HTML]{FFFFFF}1.8 V} \\
\multicolumn{1}{|c|}{\cellcolor[HTML]{FFFFFF}$V_{read}$} & \multicolumn{1}{l}{\cellcolor[HTML]{FFFFFF}read voltage} & \multicolumn{1}{r|}{\cellcolor[HTML]{FFFFFF}0.5 V} \\
\multicolumn{1}{|c|}{\cellcolor[HTML]{FFFFFF}$V_{write}$} & \multicolumn{1}{l}{\cellcolor[HTML]{FFFFFF}write voltage} & \multicolumn{1}{r|}{\cellcolor[HTML]{FFFFFF}1.2 V} \\
\multicolumn{1}{|c|}{\cellcolor[HTML]{FFFFFF}$t_{read}$} & \multicolumn{1}{l}{\cellcolor[HTML]{FFFFFF}current-read out time} & \multicolumn{1}{r|}{\cellcolor[HTML]{FFFFFF}10 ns} \\
\multicolumn{1}{|c|}{\cellcolor[HTML]{FFFFFF}$t_{write}$} & \multicolumn{1}{l}{\cellcolor[HTML]{FFFFFF}programming time} & \multicolumn{1}{r|}{\cellcolor[HTML]{FFFFFF}250 ns} \\ 
\multicolumn{1}{|c|}{\cellcolor[HTML]{FFFFFF}$n$} & \multicolumn{1}{l}{\cellcolor[HTML]{FFFFFF}number of memristors} & \multicolumn{1}{r|}{\cellcolor[HTML]{FFFFFF}200} \\
\multicolumn{1}{|c|}{\cellcolor[HTML]{FFFFFF}$v_{Th}$} & \multicolumn{1}{l}{\cellcolor[HTML]{FFFFFF}threshold voltage} & \multicolumn{1}{r|}{\cellcolor[HTML]{FFFFFF}$|$0.4$|$ V} \\ 
\multicolumn{1}{|c|}{\cellcolor[HTML]{FFFFFF}$P_{critical}$} & \multicolumn{1}{l}{\cellcolor[HTML]{FFFFFF}worst case power consumption} & \multicolumn{1}{r|}{\cellcolor[HTML]{FFFFFF}2.9 mW} \\ 
\multicolumn{1}{|c|}{\cellcolor[HTML]{FFFFFF}$R_{wire}$} & \multicolumn{1}{l}{\cellcolor[HTML]{FFFFFF}wire resistance} & \multicolumn{1}{r|}{\cellcolor[HTML]{FFFFFF}3.2$\Omega$ p/cell} \\ 
\multicolumn{1}{|c|}{\cellcolor[HTML]{FFFFFF}$A_{cell}$} & \multicolumn{1}{l}{\cellcolor[HTML]{FFFFFF}cell area} & \multicolumn{1}{r|}{\cellcolor[HTML]{FFFFFF}20 $\mu$m $\times$ 20$\mu$m} \\ 
\multicolumn{1}{|c|}{\cellcolor[HTML]{FFFFFF}$W/L$} & \multicolumn{1}{l}{\cellcolor[HTML]{FFFFFF}transistor sizing} & \multicolumn{1}{r|}{\cellcolor[HTML]{FFFFFF}450nm/180nm = 2.5} \\ \hline \vspace{-6pt}
 &  &  \\ \hline
\end{tabular}%
\end{table}

\subsection{Cell Test}


\begin{figure*}
\centering
\subfloat[]
{
	\includegraphics[scale=.47]{Fig4a}
	\label{fig:5a}
}
\subfloat[]
{
	\includegraphics[scale=.47]{Fig4b}
	\label{fig:5b}
}
\subfloat[]
{
	\includegraphics[scale=.28]{Fig5a}
	\label{fig:5a}
	}
	\subfloat[]
{
	\includegraphics[scale=.28]{Fig5b}
	\label{fig:5a}
	}\\
%\subfloat[]
%{
%	\includegraphics[scale=.4]{Fig5c}
%	\label{fig:5c}
%}
\caption{Experimental results (a) Pinched hysteresis loop of a single memristor (b) IR loss comparison in expansion mode (c) Worst-case leakage current through transistor \texttt{N1} during a write cycle in deep-net mode with Monte Carlo parameter sweep of memristance (d) Extreme test case under large input write signal resulting in a nonlinear voltage drop across the memristor in deep-net mode.}
\label{fig:4}
\end{figure*}

The CMOS cell was designed in the SK Hynix 180-nm process where $v_{DD}$ = 1.8 V, $v_{Th}$ = 0.4 V, and our parameters are summarized in Table I. We used a read voltage in the range of \{0V, 0.5V\}, a write voltage of 1.2~V, both applied at $V_{IN}$, with measurements taken with a Micromanipulator tungsten probe tip. The pinched hysteresis loop measured with a 50Hz source is shown in Fig.~3(a). 

First, we tested the circuit in expansion mode. In the critical case of a write voltage applied to all devices $V_{write}$=1.2V and \texttt{RE} is set \texttt{HIGH} (1.8V), we show that IR drop is decreased by approximately 22\% compared to a similar planar inference engine in \cite{Hu2016}. This is shown by the slower decline of current output across columns in Fig.~3(b) for CrossStack, where the gold standard would be a perfectly straight line. Therefore, we verified that expansion mode reduces line losses for a given number of inputs due to the shorter length of column wire required. The trade-off is that column wires must handle twice the current capacity of an equivalent 2-D array. This demands wide column lines to handle such current capacity without risk of electromigration. But given that RRAM is integrated in the back end of the line, minimum thickness of higher layer routing wires may mitigate this.

Designing for deep-net mode opens up susceptibility to leakage currents through \texttt{N1} during write mode, concurrently with \texttt{N2} in read mode (see Fig.~1). The worst case leakage occurs along the shared column line, when there is a minimal read current and a maximal write current. The read array memristors will all be \texttt{OFF}, $R_r = 100K\Omega$, and all write array memristors will be \texttt{ON}, $R_s=10K\Omega$. To calculate the minimum value for $V_{read}$, we use 0.5~V as the maximum read voltage and assume an input of 7-bit resolution which requires increments of approximately $500mV/128 \approx 4mV$. The output current of a single cell under these conditions was measured to be 39.6~nA, which is 1\% off the ideal 40~nA. The accumulated leakage current through a column of 10 memristors being programmed ($V_{IN}$=1.2V; \texttt{RE=LOW}) was negligible in our experiments, and simulated to be approximately 2.5~pA per cell (i.e., 2.5pA$\times$10 cells = 25pA column current). This is $6.3\times10^{-2}$\% of the worst-case read-current, and so in the 180-nm process used, we are able to employ minimum transistor dimensions (W = 450nm, L = 180nm, W/L = 2.5). Leakage from a single cell is shown as a function of a DC sweep at the input is measured in Fig.~3(c) and (d), with a Monte Carlo parametric sweep of resistance overlaid (10k$\Omega\pm$7\%, Gaussian distribution across 200 trials).

\begin{figure}
\centering
\includegraphics[scale=0.6]{fig5c.png}
\caption{Transient analysis of a single cell current output during deep-net mode in a read cycle.}
\end{figure}

Current read-out measurements taken from a transient analysis under a switching input during a read-cycle are given in Fig.~4, where current deviates by 8\% from the measured value in the worst-case. This suggests that a single device should be limited to 3.5-bits. We note this is a limitation not of the CrossStack architecture, but rather of memristor variability. In the conservative case of a 1-bit cell across 10 columns, the ~10ns read-out is subsumed within the 25ns programming time as opposed to being added to a separate cycle after programming. This confirms improvement in speed due to our in-situ pipelining mechanism by 29\%. 

%Expansion Mode: in the critical scenario of VIN = 1.8V, and all memristors are set to RS = 10KΩ we measured the IR drop to be 9.61\% of the input at the final column (20 MRs).An equivalent 2-D crossbar was simulated with an IR drop of 31.67\%.
% The  read-out current in Fig. 5(b) when R = RS = 1KΩ.

%\subsection{Convolutional Example}

%To verify functionality of CrossStack, we performed a 2D convolution using all 10 columns and 9 rows across on a 256 × 256 8-bit unsigned representaiton of a photograph of Shibuya Crossing in Tokyo. Each output pixel value is mapped from a current read-out of the chip processed via MATLAB in expansion mode. A 3 × 3 Gaussian blur smoothing kernel was applied: [1 2 1 ; 2 4 2; 1 2 1], where a value of ‘1’ required the pair of stacked MRs to both be off, a value of ‘4’ had both MRs on, and ‘2’ had one on and one off. The input image is convolved with a stride of one without zero-padding, so the output image is 254 × 254 across 64,516 convolutions for each RGB channel (Fig. 7).

\section{Discussion and Conclusion}
%WHY WOULD YOU USE EXPANSION? WHY WOULD YOU USE DEEP-NET??
With two different modes available to the user, how would you decide to use one over another? In short, deep-net is suited for tasks where speed is paramount and expansion mode is for tasks involving a large number of inputs, be it for increased resolution, or to process a very large number of parameters in a shallow network.

There are three primary motivations in the use of expansion mode: 1) fully-connected networks typically have a far larger number of connections than in convolutional layers. Expansion mode doubles the number of possible inputs for a fixed area which enables increases the number of possible inputs; 2) a larger number of inputs requires a larger length of column wire. By using expansion mode to double up on inputs, we reduce wire resistance to half the original amount for a given number of inputs. More inputs per unit length of wire enables our crossbar to be resilient to write failures arising from line resistance IR drops by reducing line losses by 22\%. 3) 
%The third motivation addresses the issue with analog computing in neural network chips. Analog computation is appealing at first, in that a single 64-level memristor can compute the same amount of information as a row of 6~$\times$~2-bit memristors. But the fundamental lower limit of energy consumption for D/A conversion counteracts the advantages (unless absolutely everything is done in analog: sensor, pre-processing, inference). 
The lack of reliable analog memory technology makes it hard to perform hardware multiplexing in analog, and transmitting analog values over long distances or at high speed is not efficient. Restricting each memristor to one of two conductance values (i.e., single-bit memristors) means that one would require $log_2(n)$ memristors for $n$ bits of precision. For crossbars that use conductance states of memristors conservatively, digital computing is more desirable but requires more memristors in a crossbar than analog for the same precision. Expansion mode facilitates this increase in devices whilst halving crossbar area. The drawback is that a larger current must be carried through the wire with more vias.

Deep-net mode is a novel processing scheme where the two crossbar layers are isolated from one another by appropriately biasing \texttt{RE}, in order to parallelize read and write operations. It is engaged where speed is of greater importance than precision. In the most simplistic way (ignoring max-pooling and dropout), a crossbar performs inference in the following way:
\begin{enumerate}
\item Write weights to memristor conductances,
\item Apply a sub-threshold read voltage,
\item Buffer or store read-out signal in memory,
\item Write the next hidden layer weights to the crossbar,
\item Repeat steps 2-5 until output is generated.
\end{enumerate}

\noindent In deep-net mode, CrossStack performs steps 2 and 4 simultaneously which enables read and write operations to occur together. By the time an output is generated from step 3, it is ready to be processed by the next hidden layer in step 4 for a speed increase of 29\% over an equivalent 2-D array. 

The most prevailing issues in realizing large scale crossbar arrays beyond our working prototype are mismatch and endurance. At this stage, our current error rate was 8\% which limits the number of bits that can be represented by a single cell. Subthreshold current was hardly an issue in our design, but Fig.~3(d) shows the nonlinearity of voltage across the memristor under high write voltages ($V_{IN}>3V$), and leakages will become more prevalent at shorter channel lengths. Capacitive coupling as a result of high programming voltages into bit-lines that are reading out poses a degree of risk, but should be tolerable for higher metal layers given the wider spacing. Allowing for heat dissipation in a 3D structure is an ongoing challenge and the subject of significant process-related research, often calling for external heat sinks. In general, the advantages seem to outweigh the drawbacks and CrossStack presents a promising methodology for reconfigurable inference acceleration to adapt to the various types of ANNs being deployed.


\section*{Acknowledgment}
This work was supported by the National Research Foundation of Korea grant funded by the Korean government (MSIT) (No.2020R1F1A1069381).

%The authors would like to thank...


% Can use something like this to put references on a page
% by themselves when using endfloat and the captionsoff option.
%\ifCLASSOPTIONcaptionsoff
%  \newpage
%\fi



% trigger a \newpage just before the given reference
% number - used to balance the columns on the last page
% adjust value as needed - may need to be readjusted if
% the document is modified later
%\IEEEtriggeratref{8}
% The "triggered" command can be changed if desired:
%\IEEEtriggercmd{\enlargethispage{-5in}}

% references section

% can use a bibliography generated by BibTeX as a .bbl file
% BibTeX documentation can be easily obtained at:
% http://mirror.ctan.org/biblio/bibtex/contrib/doc/
% The IEEEtran BibTeX style support page is at:
% http://www.michaelshell.org/tex/ieeetran/bibtex/
% argument is your BibTeX string definitions and bibliography database(s)
%\bibliography{IEEEabrv,../bib/paper}
%
% <OR> manually copy in the resultant .bbl file
% set second argument of \begin to the number of references
% (used to reserve space for the reference number labels box)

\begin{thebibliography}{l}



\bibitem{Mahajan2018}
D.~Mahajan, R.~Girshick, V.~Ramanathan, K.~He, M.~Paluri, Y.~Li, A.~Bharambe and L.~van~der~Maaten, ``Exploring the limits of weakly supervised pretraining,'' \emph{Proc. of the European Conf. on Computer Vision (EECV)}, pp.~181--196, 2018.

A.~G.~Howard, \textit{et al.}, ``MobileNets: Efficient convolutional neural networks for mobile vision applications'', arXiv preprint, arXiv:1704.04861.
%\bibitem{Li2016}
%H.~Li, A.~Kadav, I.~Durdanovic, H.~Samet and H.~P.~Graf, ``Pruning filters for efficient convnets'', \emph{arXiv preprint arXiv:1608.08710}, Aug. 2016.


%\bibitem{Frankle2019}
%J.~Frankle and M.~Carbin, ``The lottery ticket hypothesis: Finding sparse, trainable neural networks'', \emph{International Conference on Learning Representations (ICLR)}, May 2019.


%\bibitem{Azevedo2009}
%F.~A.~Azevedo, L.~R.~Carvalho, L.~T.~Grinberg, J.~M.~Farfel, R.~E.~Ferretti, R.~E.~Leite, W.~J.~Filho, R.~Lent and S.~Herculano-Houzel, ``Equal numbers of neuronal and nonneuronal cells make the human brain an isometrically scaled-up primate brain,'' \emph{Journal of Comparative Neurology}, vol.~513, no.~5, pp.~532--541, Apr. 2009.


\bibitem{Kim2010}
K.~H.~Kim, S.~Jo, S.~Gaba and W.~Lu, ``Nanoscale resistive memory with intrinsic diode characteristics and long endurance,'' \emph{Applied Physics Letters}, vol.~96, no.~5, p.~053106, Feb. 2010.

\bibitem{Kumar2017}
A.~Kumar, M.~Das, V.~Garg, B.~S.~Sengar, M.~T.~Htay, S.~Kumar, A.~Kranti and S.~Mukherjee, ``Forming-free high-endurance Al/ZnO/Al memristor fabricated by dual ion beam sputtering,'' \emph{Applied Physics Letters}, vol~110, no.~25, p.~253509, Jun.~2017.

\bibitem{Lin20201}
C.-Y.~Lin \textit{et al.}, ``Adaptive synaptic memory via lithium ion modulation in RRAM devices'', \emph{Small}, vol.~16, no.~42, p.~2003964, Oct. 2020. 

\bibitem{Pi2019}
S.~Pi, C.~Li, H.~Jiang, W.~Xia, H.~Xin, J.~J.~Yang and Q.~Xia, ``Memristor crossbar arrays with 6-nm half-pitch and 2-nm critical dimension,'' \emph{Nature nanotechnology}, vol.~14, no.~1, p.~35, Jan.~2019.

\bibitem{Fuller2019}
E.~J.~Fuller, S.~T.~Keene, A.~Melianas, Z.~Wang, S.~Agarwal, Y.~Li, Y.~Tuchman, C.~D.~James, M.~J.~Marinella, J.~J.~Yang, A.~salleo and A.~A.~Talin, ``Parallel programming of an ionic floating-gate memory array for scalable neuromorphic computing'', \emph{Science}, vol. 364, no. 6440, pp. 570--574, May 2019.


\bibitem{Merced2016}
E.~J.~Merced-Grafals, N.~Dávila, N.~Ge, R.~S.~Williams and J.~P.~Strachan, ``Repeatable, accurate, and high speed multi-level programming of memristor 1T1R arrays for power efficient analog computing applications,'' \emph{Nanotechnology}, vol.~27, no.~36, p.~365202, Aug.~2016.

\bibitem{Lin20202}
C.-Y.~Lin \textit{et al.}, ``A high-speed MIM resistive memory cell with an inherent vanadium selector'', \emph{Applied Materials Today}, vol.~21, p.~100848, Dec. 2020.

\bibitem{Eshraghian2018}
J.~K.~Eshraghian, K.~Cho, C.~Zheng, M.~Nam, H.~H.~C.~Iu, W.~Lei, and K.~Eshraghian, ``Neuromorphic Vision Hybrid RRAM-CMOS Architecture,'' \emph{IEEE Trans. Very Large Scale Integration (VLSI) Systems}, vol. 26, no. 12, pp. 2816–-2829, Dec. 2018

\bibitem{Chakrabarti2017}
B.~Chakrabarti, M.~A.~Lastras-Montaño, G.~Adam, M.~Prezioso, B.~Hoskins, M.~Payvand, A.~Madhavan, A.~Ghofrani, L.~Theogarajan, K.~T.~Cheng, D.~B.~Strukov, ``A multiply-add engine with monolithically integrated 3D memristor crossbar/CMOS hybrid circuit,'' \emph{Scientific reports}, vol.~14, no.~7, p.~42429, Feb. 2017.

\bibitem{Cai2019}
F.~Cai, J.~M.~Correll, S.~H.~Lee, Y.~Lim, V.~Bothra, Z.~Zhang, M.~P.~Flynn and W.~D.~Lu, ``A fully integrated reprogrammable memristor-CMOS system for efficient multiply-accumulate operations'', \emph{Nature Electronics}, vol.~15, no.~1, Jul.~2019.

\bibitem{Azghadi2020}
M.~R.~Azghadi \textit{et al.}, ``Complementary Metal-Oxide Semiconductor and Memristive Hardware for Neuromorphic Computing'', \emph{Advanced Intelligent Systems}, vol.~2, no.~5, Mar.~2020.


\bibitem{Lammie2020}
C.~Lammie and M.~R.~Azghadi, ``MemTorch: A simulation framework for deep memristive Cross-Bar architectures'', \emph{2020 IEEE Int. Symp. Circuits and Systems (ISCAS), Sevilla, Spain, Oct.~2020.}

\bibitem{Krestinskaya2018}
O.~Krestinskaya, K.~N.~Salama and A.~P.~James, ``Analog Backpropagation Learning Circuits for Memristive Crossbar Neural Networks'', \emph{2018 IEEE Int. Symp. Circuits and Systems (ISCAS), Florence, Italy, May 2018}

\bibitem{Serb2016}
A.~Serb, J.~Bill, A.~Khiat, R.~Berdan, R.~Legenstein and T.~Prodromakis, ``Unsupervised learning in probabilistic neural networks with multi-state metal-oxide memristive synapses,'' \emph{Nature communications}, vol.~7, p.~12611, Sep.~2016.

\bibitem{Chevallier2010}
C.~J.~Chevallier, C.~H.~Siau, S.~F.~Lim, S.~R.~Namala, M.~Matsuoka, B.~L.~Bateman and D.~A.~Rinerson, ``A 0.13$\mu$m 64Mb multi-layered conductive metal-oxide memory'', \emph{2010 IEEE Int. Solid-State Circuits Conf. (ISSCC)}, pp.~260--261, Feb.~2010.

\bibitem{Wang2011}
C.~H.~Wang, Y.~H.~Tsai, K.~C.~Lin, M.~F.~Chang, Y.~C.~King, C.~J.~Lin, S.~S.~Sheu, Y.~S.~Chen, Y.~H.~Lee, F.~T.~Chen and M.~J.~Tsai, ``Three-dimensional $4F^2$ ReRAM cell with vertical BJT driver by CMOS logic compatible process,'' \emph{IEEE Trans. on Electron Devices}, vol.~58, no.~8, pp.~2466--2472, Jul~2011.

\bibitem{Jo2014}
S.~H.~Jo, T.~Kumar, S.~Narayanan, W.~D.~Lu and H.~Nazarian, ``3D-stackable crossbar resistive memory based on field assisted superlinear threshold (FAST) selector,'' \emph{2014 IEEE Int. Electron Devices Meeting}, Washington, DC, USA, pp.~6--7, Dec.~2014.

\bibitem{Hong2018}
J.~Hong, M.~Stone, B.~Navarrete, K.~Luongo, Q.~Zheng, Z.~Yuan, K.~Xia, N.~Xu, J.~Bokor, L.~You and S.~Khizroev, ``3D multilevel spin transfer torque devices,'' \emph{Applied Physics Letters}, vol.~112, no.~11, p.~112402, Mar. 2018.

\bibitem{Azghadi20201}
M.~R.~Azghadi \textit{et al.}, ``Hardware implementation of deep network accelerators towards healthcare and biomedical applications'', \emph{IEEE Trans. Biomedical Circuits and Systems}, vol.~14, no.~6, pp.~1138-1159, Dec. 2020.

\bibitem{Baek2019}
S.~Baek, J.~K.~Eshraghian, S.-H.~Ahn, A.~James and K.~Cho, ``A memristor-CMOS Braun multiplier array for arithmetic pipelining'', \emph{2019 26th IEEE Int. Conf. Electronics, Circuits and Systems (ICECS), Genoa, Italy, pp. 735--738, Nov. 2019.}

\bibitem{Fastow2019}
R.~Fastow, K.~Hasnat, P.~Majhi and O.~Jungroth, ``Three-dimensional (3D) memory with shared control circuitry using wafer-to-wafer bonding,'' \emph{United States Patent Application}, US 16/011, 139, Feb.~2019.

\bibitem{Eshraghian2017}
J.~K.~Eshraghian, K.~R.~Cho, H.~H.~C.~Iu, T.~Fernando, N.~Iannella, S.~M.~Kang and K.~Eshraghian, ``Maximization of crossbar array memory using fundamental memristor theory,'' \emph{IEEE Trans. on Circuits and Syst. II: Express Briefs}, vol.~64, no.~12, pp.~1402--1406, Oct. 2017.

\bibitem{Li2018}
C.~Li, M.~Hu, Y.~Li, H.~Jiang, N.~Ge, E.~Montgomery, J.~Zhang, W.~Song, N.~Davila, C.~E.~Graves, Z.~Li, J.~P.~Strachan, P.~Lin, Z.~Wang, M.~Barnell, Q.~Wu, R.~S.~Williams, J.~J.~Yang, and Q.~Xia, ``Analogue signal and image processing with large memristor crossbars'', \emph{Nature Electronics}, vol. 1, no. 1, pp. 52-–59, Jan. 2018.

\bibitem{Eshraghian2019}
J.~K.~Eshraghian, S.~M.~Kang, S.~Baek, G.~Orchard, H.~H.~C.~Iu, and W.~Lei, ``Analog weights in ReRAM DNN Accelerators'', \emph{2019 IEEE Artificial Circuits and Systems Conference}, Mar. 2019.

\bibitem{Hu2016}
M.~Hu, J.~P.~Strachan, Z.~Li, E.~M.~Grafals, N.~Davila, C.~Graves, S.~Lam, N.~Ge, J.~J.~Yang, R.~S.~Williams, ``Dot-product engine for neuromorphic computing: Programming 1T1M crossbar to accelerate matrix-vector multiplication,'' \emph{Proc. of the 53rd Annual Design Automation Conference}, p.~19, Jun. 2016.

\bibitem{Shafiee2016}
A.~Shafiee, A.~Nag, N.~Muralimanohar, R.~Balasubramonian, J.~P.~Strachan, M.~Hu, R.~S.~Williams and V.~Srikumar, ``ISAAC: A convolutional neural network accelerator with in-situ analog arithmetic in crossbars,'' \emph{ACM SIGARCH Computer Architecture News}, vol.~44, no.~3, pp.~14--26, Oct. 2016.

\bibitem{Stathopoulos2017}
S.~Stathopoulos, A.~Khiat, M.~Trapatseli, S.~Cortese, A.~Serb, I.~Valov and T.~Prodromakis, ``Multibit memory operation of metal-oxide bi-layer memristors,'' \emph{Scientific Reports}, vol.~7, no.~17532, Dec. 2017.

\bibitem{Song2017}
L.~Song, X.~Qian, H.~Li and Y.~Chen, ``Pipelayer: A pipelined ReRAM-based accelerator for deep learning,'' \emph{2017 IEEE Int. Symp. on High Performance Computer Architecture (HPCA)}, pp.~541--552, Feb.~2017.

\bibitem{Valavi2019}
H.~Valavi, P.~J.~Ramadge, E.~Nestler and N.~Verma, ``A 64-Tile 2.4-Mb In-Memory-Computing CNN accelerator employing charge-domain computer,'' \emph{IEEE Journal of Solid-State Circuits}, vol.~54, no.~6, pp.~1789-1799, Mar.~2019.

\bibitem{Adam2017}
G.~C.~Adam, B.~D.~Hoskins, M.~Prezioso, F.~Merrikh-Bayat, B.~Chakrabarti and D.~B.~Strukov, ``3-D memristor crossbars for analog and neuromorphic computing applications,'' \emph{IEEE Transactions on Electron Devices}, vol.~64, no.~1, pp.~312--318, Jan.~2017.

%\bibitem{Lee2011}
%M.~-J.~Lee, C.~B.~Lee, D.~Lee, S.~R.~Lee, M.~Chang, J.~H.~Hur, Y.~Kim, C.~Kim, D.~H.~Seo, S.~Seo, U.~Chung, I.~Yoo, and K.~Kim, ``A fast, high-endurance and scalable non-volatile memory device made from asymmetric \ce{Ta2O_{5-−x}}/\ce{TaO_{2-−x}} bilayer structures,'' \emph{Nature Materials}, vol. 10, pp. 625-–630, Jul. 2011.

%\bibitem{Torrezan2011}
%A.~C.~Torrezan, J.~P.~Strachan, G.~Medeiros-Ribeiro, and R.~S.~Williams, ``Sub-nanosecond switching of a tantalum oxide memristor'', \emph{Nanotechnology}, vol. 22, no. 48, p. 485203 Nov. 2011.

%\bibitem{Murdoch2016}
%B.~J.~Murdoch, D.~G.~McCulloch, R.~Ganesan, D.~R.~McKenzie, M.~M.~M.~Bilek, and J.~G.~Partridge, ``Memristor and selector devices fabricated from \ce{HfO_{2−-x}N_x}'', \emph{Applied Physics Letters}, vol. 108, p. 143504, Apr. 2016.

%\bibitem{Strukov2008}
%D.~B.~Strukov, G.~S.~Snider, D.~R.~Stewart, and R.~S.~Williams, ``The missing memristor found'', \emph{Nature}, vol. 453, pp. 80-–83, May 2008.

%\bibitem{Yang2008}
%J.~J.~Yang, M.~D.~Pickett, X.~Li, D.~A.~A.Ohlberg, D.~R.~Stewart, and R.~S.~Williams, ``Memristive switching mechanism for metal/oxide/metal nanodevices,'' \emph{Nature Nanotechnology}, vol. 3, pp. 429-–433, Jun. 2008.

%\bibitem{Kwon2010}
%D.~Kwon, K.~M.~Kim, J.~H.~Jang, J.~M.~Jeon, M.~H.~Lee, G.~H.~Kim, X.~Li, G.~Park, B.~Lee, S.~Han, M.~Kim, and C.~S.~Hwang, ``Atomic structure of conducting nanofilaments in \ce{TiO2} resistive switching memory,'' \emph{Nature Nanotechnology}, vol. 5, pp. 148-–153, Jan. 2010.

%\bibitem{Liu2015}
%C.~Liu, B.~Yan, C.~Yang, L.~Song, Z.~Li, B.~Liu, Y.~Chen, H.~Li, Q.~Wu, and H.~Jiang, ``A spiking neuromorphic design with resistive crossbar,'' \emph{IEEE ACM/EDAC/IEEE Design Automation Conf. (DAC)}, pp. 1–6, Jun. 2015.


%\bibitem{Zhu2019}
%X.~Zhu, S.~H.~Lee, W.~D.~Lu, ``Nanoionic resistive-switching devices',' \emph{Advanced Electronic Materials}, p.~1900184, May 2019.

\end{thebibliography}

\end{document}



2102.06536.extracted_bib
{Mahajan2018 D.~Mahajan, R.~Girshick, V.~Ramanathan, K.~He, M.~Paluri, Y.~Li, A.~Bharambe and L.~van~der~Maaten, ``Exploring the limits of weakly supervised pretraining,'' Proc. of the European Conf. on Computer Vision (EECV), pp.~181--196, 2018. A.~G.~Howard, et al., ``MobileNets: Efficient convolutional neural networks for mobile vision applications'', arXiv preprint, arXiv:1704.04861. %
2102.06536.extracted_bib
{Li2016 %H.~Li, A.~Kadav, I.~Durdanovic, H.~Samet and H.~P.~Graf, ``Pruning filters for efficient convnets'', arXiv preprint arXiv:1608.08710, Aug. 2016. %
2102.06536.extracted_bib
{Frankle2019 %J.~Frankle and M.~Carbin, ``The lottery ticket hypothesis: Finding sparse, trainable neural networks'', International Conference on Learning Representations (ICLR), May 2019. %
2102.06536.extracted_bib
{Azevedo2009 %F.~A.~Azevedo, L.~R.~Carvalho, L.~T.~Grinberg, J.~M.~Farfel, R.~E.~Ferretti, R.~E.~Leite, W.~J.~Filho, R.~Lent and S.~Herculano-Houzel, ``Equal numbers of neuronal and nonneuronal cells make the human brain an isometrically scaled-up primate brain,'' Journal of Comparative Neurology, vol.~513, no.~5, pp.~532--541, Apr. 2009.
2102.06536.extracted_bib
{Kim2010 K.~H.~Kim, S.~Jo, S.~Gaba and W.~Lu, ``Nanoscale resistive memory with intrinsic diode characteristics and long endurance,'' Applied Physics Letters, vol.~96, no.~5, p.~053106, Feb. 2010.
2102.06536.extracted_bib
{Kumar2017 A.~Kumar, M.~Das, V.~Garg, B.~S.~Sengar, M.~T.~Htay, S.~Kumar, A.~Kranti and S.~Mukherjee, ``Forming-free high-endurance Al/ZnO/Al memristor fabricated by dual ion beam sputtering,'' Applied Physics Letters, vol~110, no.~25, p.~253509, Jun.~2017.
2102.06536.extracted_bib
{Lin20201 C.-Y.~Lin et al., ``Adaptive synaptic memory via lithium ion modulation in RRAM devices'', Small, vol.~16, no.~42, p.~2003964, Oct. 2020.
2102.06536.extracted_bib
{Pi2019 S.~Pi, C.~Li, H.~Jiang, W.~Xia, H.~Xin, J.~J.~Yang and Q.~Xia, ``Memristor crossbar arrays with 6-nm half-pitch and 2-nm critical dimension,'' Nature nanotechnology, vol.~14, no.~1, p.~35, Jan.~2019.
2102.06536.extracted_bib
{Fuller2019 E.~J.~Fuller, S.~T.~Keene, A.~Melianas, Z.~Wang, S.~Agarwal, Y.~Li, Y.~Tuchman, C.~D.~James, M.~J.~Marinella, J.~J.~Yang, A.~salleo and A.~A.~Talin, ``Parallel programming of an ionic floating-gate memory array for scalable neuromorphic computing'', Science, vol. 364, no. 6440, pp. 570--574, May 2019.
2102.06536.extracted_bib
{Merced2016 E.~J.~Merced-Grafals, N.~Dávila, N.~Ge, R.~S.~Williams and J.~P.~Strachan, ``Repeatable, accurate, and high speed multi-level programming of memristor 1T1R arrays for power efficient analog computing applications,'' Nanotechnology, vol.~27, no.~36, p.~365202, Aug.~2016.
2102.06536.extracted_bib
{Lin20202 C.-Y.~Lin et al., ``A high-speed MIM resistive memory cell with an inherent vanadium selector'', Applied Materials Today, vol.~21, p.~100848, Dec. 2020.
2102.06536.extracted_bib
{Eshraghian2018 J.~K.~Eshraghian, K.~Cho, C.~Zheng, M.~Nam, H.~H.~C.~Iu, W.~Lei, and K.~Eshraghian, ``Neuromorphic Vision Hybrid RRAM-CMOS Architecture,'' IEEE Trans. Very Large Scale Integration (VLSI) Systems, vol. 26, no. 12, pp. 2816–-2829, Dec. 2018
2102.06536.extracted_bib
{Chakrabarti2017 B.~Chakrabarti, M.~A.~Lastras-Montaño, G.~Adam, M.~Prezioso, B.~Hoskins, M.~Payvand, A.~Madhavan, A.~Ghofrani, L.~Theogarajan, K.~T.~Cheng, D.~B.~Strukov, ``A multiply-add engine with monolithically integrated 3D memristor crossbar/CMOS hybrid circuit,'' Scientific reports, vol.~14, no.~7, p.~42429, Feb. 2017.
2102.06536.extracted_bib
{Cai2019 F.~Cai, J.~M.~Correll, S.~H.~Lee, Y.~Lim, V.~Bothra, Z.~Zhang, M.~P.~Flynn and W.~D.~Lu, ``A fully integrated reprogrammable memristor-CMOS system for efficient multiply-accumulate operations'', Nature Electronics, vol.~15, no.~1, Jul.~2019.
2102.06536.extracted_bib
{Azghadi2020 M.~R.~Azghadi et al., ``Complementary Metal-Oxide Semiconductor and Memristive Hardware for Neuromorphic Computing'', Advanced Intelligent Systems, vol.~2, no.~5, Mar.~2020.
2102.06536.extracted_bib
{Lammie2020 C.~Lammie and M.~R.~Azghadi, ``MemTorch: A simulation framework for deep memristive Cross-Bar architectures'', 2020 IEEE Int. Symp. Circuits and Systems (ISCAS), Sevilla, Spain, Oct.~2020.
2102.06536.extracted_bib
{Krestinskaya2018 O.~Krestinskaya, K.~N.~Salama and A.~P.~James, ``Analog Backpropagation Learning Circuits for Memristive Crossbar Neural Networks'', 2018 IEEE Int. Symp. Circuits and Systems (ISCAS), Florence, Italy, May 2018
2102.06536.extracted_bib
{Serb2016 A.~Serb, J.~Bill, A.~Khiat, R.~Berdan, R.~Legenstein and T.~Prodromakis, ``Unsupervised learning in probabilistic neural networks with multi-state metal-oxide memristive synapses,'' Nature communications, vol.~7, p.~12611, Sep.~2016.
2102.06536.extracted_bib
{Chevallier2010 C.~J.~Chevallier, C.~H.~Siau, S.~F.~Lim, S.~R.~Namala, M.~Matsuoka, B.~L.~Bateman and D.~A.~Rinerson, ``A 0.13$\mu$m 64Mb multi-layered conductive metal-oxide memory'', 2010 IEEE Int. Solid-State Circuits Conf. (ISSCC), pp.~260--261, Feb.~2010.
2102.06536.extracted_bib
{Wang2011 C.~H.~Wang, Y.~H.~Tsai, K.~C.~Lin, M.~F.~Chang, Y.~C.~King, C.~J.~Lin, S.~S.~Sheu, Y.~S.~Chen, Y.~H.~Lee, F.~T.~Chen and M.~J.~Tsai, ``Three-dimensional $4F^2$ ReRAM cell with vertical BJT driver by CMOS logic compatible process,'' IEEE Trans. on Electron Devices, vol.~58, no.~8, pp.~2466--2472, Jul~2011.
2102.06536.extracted_bib
{Jo2014 S.~H.~Jo, T.~Kumar, S.~Narayanan, W.~D.~Lu and H.~Nazarian, ``3D-stackable crossbar resistive memory based on field assisted superlinear threshold (FAST) selector,'' 2014 IEEE Int. Electron Devices Meeting, Washington, DC, USA, pp.~6--7, Dec.~2014.
2102.06536.extracted_bib
{Hong2018 J.~Hong, M.~Stone, B.~Navarrete, K.~Luongo, Q.~Zheng, Z.~Yuan, K.~Xia, N.~Xu, J.~Bokor, L.~You and S.~Khizroev, ``3D multilevel spin transfer torque devices,'' Applied Physics Letters, vol.~112, no.~11, p.~112402, Mar. 2018.
2102.06536.extracted_bib
{Azghadi20201 M.~R.~Azghadi et al., ``Hardware implementation of deep network accelerators towards healthcare and biomedical applications'', IEEE Trans. Biomedical Circuits and Systems, vol.~14, no.~6, pp.~1138-1159, Dec. 2020.
2102.06536.extracted_bib
{Baek2019 S.~Baek, J.~K.~Eshraghian, S.-H.~Ahn, A.~James and K.~Cho, ``A memristor-CMOS Braun multiplier array for arithmetic pipelining'', 2019 26th IEEE Int. Conf. Electronics, Circuits and Systems (ICECS), Genoa, Italy, pp. 735--738, Nov. 2019.
2102.06536.extracted_bib
{Fastow2019 R.~Fastow, K.~Hasnat, P.~Majhi and O.~Jungroth, ``Three-dimensional (3D) memory with shared control circuitry using wafer-to-wafer bonding,'' United States Patent Application, US 16/011, 139, Feb.~2019.
2102.06536.extracted_bib
{Eshraghian2017 J.~K.~Eshraghian, K.~R.~Cho, H.~H.~C.~Iu, T.~Fernando, N.~Iannella, S.~M.~Kang and K.~Eshraghian, ``Maximization of crossbar array memory using fundamental memristor theory,'' IEEE Trans. on Circuits and Syst. II: Express Briefs, vol.~64, no.~12, pp.~1402--1406, Oct. 2017.
2102.06536.extracted_bib
{Li2018 C.~Li, M.~Hu, Y.~Li, H.~Jiang, N.~Ge, E.~Montgomery, J.~Zhang, W.~Song, N.~Davila, C.~E.~Graves, Z.~Li, J.~P.~Strachan, P.~Lin, Z.~Wang, M.~Barnell, Q.~Wu, R.~S.~Williams, J.~J.~Yang, and Q.~Xia, ``Analogue signal and image processing with large memristor crossbars'', Nature Electronics, vol. 1, no. 1, pp. 52-–59, Jan. 2018.
2102.06536.extracted_bib
{Eshraghian2019 J.~K.~Eshraghian, S.~M.~Kang, S.~Baek, G.~Orchard, H.~H.~C.~Iu, and W.~Lei, ``Analog weights in ReRAM DNN Accelerators'', 2019 IEEE Artificial Circuits and Systems Conference, Mar. 2019.
2102.06536.extracted_bib
{Hu2016 M.~Hu, J.~P.~Strachan, Z.~Li, E.~M.~Grafals, N.~Davila, C.~Graves, S.~Lam, N.~Ge, J.~J.~Yang, R.~S.~Williams, ``Dot-product engine for neuromorphic computing: Programming 1T1M crossbar to accelerate matrix-vector multiplication,'' Proc. of the 53rd Annual Design Automation Conference, p.~19, Jun. 2016.
2102.06536.extracted_bib
{Shafiee2016 A.~Shafiee, A.~Nag, N.~Muralimanohar, R.~Balasubramonian, J.~P.~Strachan, M.~Hu, R.~S.~Williams and V.~Srikumar, ``ISAAC: A convolutional neural network accelerator with in-situ analog arithmetic in crossbars,'' ACM SIGARCH Computer Architecture News, vol.~44, no.~3, pp.~14--26, Oct. 2016.
2102.06536.extracted_bib
{Stathopoulos2017 S.~Stathopoulos, A.~Khiat, M.~Trapatseli, S.~Cortese, A.~Serb, I.~Valov and T.~Prodromakis, ``Multibit memory operation of metal-oxide bi-layer memristors,'' Scientific Reports, vol.~7, no.~17532, Dec. 2017.
2102.06536.extracted_bib
{Song2017 L.~Song, X.~Qian, H.~Li and Y.~Chen, ``Pipelayer: A pipelined ReRAM-based accelerator for deep learning,'' 2017 IEEE Int. Symp. on High Performance Computer Architecture (HPCA), pp.~541--552, Feb.~2017.
2102.06536.extracted_bib
{Valavi2019 H.~Valavi, P.~J.~Ramadge, E.~Nestler and N.~Verma, ``A 64-Tile 2.4-Mb In-Memory-Computing CNN accelerator employing charge-domain computer,'' IEEE Journal of Solid-State Circuits, vol.~54, no.~6, pp.~1789-1799, Mar.~2019.
2102.06536.extracted_bib
{Adam2017 G.~C.~Adam, B.~D.~Hoskins, M.~Prezioso, F.~Merrikh-Bayat, B.~Chakrabarti and D.~B.~Strukov, ``3-D memristor crossbars for analog and neuromorphic computing applications,'' IEEE Transactions on Electron Devices, vol.~64, no.~1, pp.~312--318, Jan.~2017. %
2102.06536.extracted_bib
{Lee2011 %M.~-J.~Lee, C.~B.~Lee, D.~Lee, S.~R.~Lee, M.~Chang, J.~H.~Hur, Y.~Kim, C.~Kim, D.~H.~Seo, S.~Seo, U.~Chung, I.~Yoo, and K.~Kim, ``A fast, high-endurance and scalable non-volatile memory device made from asymmetric \ce{Ta2O_{5-−x/\ce{TaO_{2-−x bilayer structures,'' Nature Materials, vol. 10, pp. 625-–630, Jul. 2011. %
2102.06536.extracted_bib
{Torrezan2011 %A.~C.~Torrezan, J.~P.~Strachan, G.~Medeiros-Ribeiro, and R.~S.~Williams, ``Sub-nanosecond switching of a tantalum oxide memristor'', Nanotechnology, vol. 22, no. 48, p. 485203 Nov. 2011. %
2102.06536.extracted_bib
{Murdoch2016 %B.~J.~Murdoch, D.~G.~McCulloch, R.~Ganesan, D.~R.~McKenzie, M.~M.~M.~Bilek, and J.~G.~Partridge, ``Memristor and selector devices fabricated from \ce{HfO_{2−-xN_x'', Applied Physics Letters, vol. 108, p. 143504, Apr. 2016. %
2102.06536.extracted_bib
{Strukov2008 %D.~B.~Strukov, G.~S.~Snider, D.~R.~Stewart, and R.~S.~Williams, ``The missing memristor found'', Nature, vol. 453, pp. 80-–83, May 2008. %
2102.06536.extracted_bib
{Yang2008 %J.~J.~Yang, M.~D.~Pickett, X.~Li, D.~A.~A.Ohlberg, D.~R.~Stewart, and R.~S.~Williams, ``Memristive switching mechanism for metal/oxide/metal nanodevices,'' Nature Nanotechnology, vol. 3, pp. 429-–433, Jun. 2008. %
2102.06536.extracted_bib
{Kwon2010 %D.~Kwon, K.~M.~Kim, J.~H.~Jang, J.~M.~Jeon, M.~H.~Lee, G.~H.~Kim, X.~Li, G.~Park, B.~Lee, S.~Han, M.~Kim, and C.~S.~Hwang, ``Atomic structure of conducting nanofilaments in \ce{TiO2 resistive switching memory,'' Nature Nanotechnology, vol. 5, pp. 148-–153, Jan. 2010. %
2102.06536.extracted_bib
{Liu2015 %C.~Liu, B.~Yan, C.~Yang, L.~Song, Z.~Li, B.~Liu, Y.~Chen, H.~Li, Q.~Wu, and H.~Jiang, ``A spiking neuromorphic design with resistive crossbar,'' IEEE ACM/EDAC/IEEE Design Automation Conf. (DAC), pp. 1–6, Jun. 2015. %
2102.06536.extracted_bib
{Zhu2019 %X.~Zhu, S.~H.~Lee, W.~D.~Lu, ``Nanoionic resistive-switching devices',' Advanced Electronic Materials, p.~1900184, May 2019.