
\section*{Acknowledgements}
This project was the result of an internship at Microsoft Research (MSR).  We
would like to thank Abhinav Shrivastava and Ishan Misra for helpful discussions
while at MSR.  We thank NVIDIA for the donation of K40 GPUs.

\section*{Appendix: 2015 MS COCO Competition}

\begin{table*}[t]
  \centering
  \input{tab_breakdown}
\end{table*}

In this section, we describe our submission to the 2015 MS COCO Detection
Challenge, which won Best Student Entry and finished $3^\text{rd}$ place
overall, with a score of 31.0\% mAP on 2015 test-challenge and 31.2\% on
2015 test-dev.  Later in this section we describe further post-competition
improvements to achieve 33.1\% on test-dev.  Both models use a single ConvNet
(no ensembling).

For our challenge submission, we made several improvements:
used a mix of MCG (Multiscale Combinatorial Grouping~\cite{MCG-PABMM2015}) and
RPN (Region Proposal Net~\cite{ren2015faster}) proposal boxes, added two extra
512x3x3 convolutional layers, trained for longer, and used two
rounds of bounding box regression with a modified version of weighted
voting~\cite{MR-CNN}.  At test time, our model runs in 2.7 seconds/image on a
single Titan X GPU (excluding proposal generation).

We describe all changes in more detail below (note that many of these choices
were driven by the need to meet the challenge deadline, and thus may be
suboptimal):
\begin{packed_enum}
  \item \textbf{Train+val.}  For the competition, we train on both train and
    validation sets.  We hold out 5000 images from the validation as our new
    validation set called ``minival.''
  \item \textbf{MCG+RPN box proposals.} We get the largest improvement by
    replacing selective search with a mix of MCG~\cite{MCG-PABMM2015} and
    RPN~\cite{ren2015faster} boxes.  We modify RPN from the baseline
    configuration described in
    \cite{ren2015faster} by adding more anchor boxes, in particular
    smaller ones, and using a mixture of 3x3 (384) and 5x5 (128) convolutions.
    Our anchor configuration uses a total of 22 anchors per location
    with the following shapes:
    32x32 and aspect ratios \{1:2, 1:1, 2:1\} $\times$ scales \{64, 90.5, 128,
    181, 256, 362, 512\}.
    We also experiment with predicting proposals from concatenated conv4\_3
    and conv5\_3 features (after L2 normalization and scaling), rather than
    from conv5\_3 only. We refer to these two configurations as RPN1
    (without concatenated features) and RPN2 (with concatenated features).
    Using VGG16, RPN1 and RPN2 achieve average recalls of 44.1\% and 44.3\%, respectively,
    compared to selective search, 41.7\%, and MCG, 51.6\%.
    Despite their lower average recalls, we found that RPN1 and RPN2 give
    comparable detection results to MCG.
    We also found that mixing 1000 MCG boxes with 1000 RPN1 or RPN2 boxes
    performs even better than 2000 of either method alone, suggesting that the
    two methods are complementary.
    Thus, we train with 1000 RPN1 and 1000 MCG boxes. At test time, we use 1000
    RPN2 and 2000 MCG boxes, which gives a +0.3 mAP improvement on minival
    compared to using the same boxes as training.
  \item \textbf{conv6+conv7.}  We use the model listed in row (c) of
    Table~\ref{tab:other}: two 512x3x3 convolutions on top of conv5\_3 which we
    call ``conv6'' and ``conv7''.  We also pool out of conv7, so in total we
    pool out of conv3\_3, conv4\_3, conv5\_3, conv7, and IRNN.
  \item \textbf{Longer training.}  Since training on COCO is very slow, we
  explored ideas by initializing new networks from the weights of the previous
  best network.  While we have not tried training our best configuration
  starting from VGG16 ImageNet weights, we do not see any reason why it would not
  work.  Here is the exact training procedure we used:
  \begin{packed_enum}
    \item Train using 2000 selective search boxes, using the schedule described
      in Section~\ref{sec:experimental-setup}, but stopping after 220k iterations
      of fine-tuning.  This achieves 24.8\% on 2015 test-dev.
    \item Change the box proposals to a mix of 1000 MCG boxes and 1000 RPN1
      boxes.  Train for 100k iterations with conv layers frozen (learning rate: exp decay
      $5 \cdot 10^{-3} \rightarrow 10^{-4}$), and for 80k iterations with
      only conv1 and conv2 frozen (learning rate: exp decay $10^{-3} \rightarrow 10^{-5}$).
  \end{packed_enum}
  \item \textbf{2xBBReg + WV.}  We use the iterative bounding box regression and
    weighted voting scheme described in \cite{MR-CNN}, but as noted in
    Section~\ref{sec:results-mscoco}, this does not work out-of-the-box for COCO
    since boxes are blurred together and precise localization is lost.  We solve
    this by adjusting the thresholds so that only very similar boxes are blurred
    together with weighted voting.  We jointly optimized the NMS (non-max
    suppression) and voting threshold on the validation set, by evaluating all
    boxes once and then randomly sampling hundreds of thresholds.  On our
    minival set, the optimal IoU (intersection-over-union) threshold is 0.443
    for NMS and 0.854 for weighted voting.  Compared to using a single round of
    standard NMS (IoU threshold 0.3), these settings give +1.3 mAP on minival.
 
   
   
   
   
   
\end{packed_enum}

\subsection*{Post-competition improvements}
We have an improved model which did not finish in time for the challenge, which
achieves 33.1\% on 2015 test-dev.  We made these further adjustments:
\begin{packed_enum}
  \item \textbf{Training longer with 0.99 momentum.}  We found that if the
    momentum vector is reset when re-starting training, the accuracy drops by
    several points and takes around 50k iterations to recover.  Based on this
    observation, we increased momentum to 0.99 and correspondingly decreased
    learning rate by a factor of 10.  We initialized from our competition
    submission (above), and:
    \begin{packed_enum}
      \item further trained for
      80k iterations with only conv1 and conv2 frozen, (learning rate: exp decay
      $10^{-4} \rightarrow 10^{-6}$).
      By itself, this gives a boost of +0.4 mAP on minival.
    \item We trained for another 160k iterations with no layers frozen (learning
      rate: exp decay $10^{-4} \rightarrow 10^{-6}$) which gives +0.7 mAP on
      minival.
  \end{packed_enum}
  \item \textbf{Left-right flipping.}  We evaluate the model twice, once on the
    original image, and once on the left-right flipped image.  To merge the
    predictions, we average both the softmax scores and box regression shifts
    (after flipping back).  By itself, this gives a boost of +0.8 mAP on minival.
  \item \textbf{More box proposals.} At test time, we use 4000 proposal boxes
    (2000 RPN2 and 2000 MCG boxes).  For the model submitted to the competition,
    this performs +0.1 mAP better than 3000 boxes (1000 RPN2 and 2000 MCG
    boxes).
\end{packed_enum}
At test time, the above model runs in 5.5 seconds/image on a single Titan X GPU
(excluding proposal generation).  Most of the slowdown is from the left-right
flipping. Table~\ref{tab:coco-breakdown} provides a breakdown of the gains due
to the various competition and post-competition changes.

\section{Architecture: Inside-Outside Net (ION)}

In this section we describe ION, a detector with an improved descriptor both
inside and outside the ROI.  An image is processed
by a single deep ConvNet, and the convolutional
feature maps at each stage of the ConvNet are stored in memory. At the top of the network,
a 2x stacked 4-directional IRNN (explained later) computes context features that
describe the image both globally and locally.  The context features have the same dimensions as
``conv5.''  This is done once per image.  In addition, we have thousands of
proposal regions (ROIs) that might contain objects.  For each ROI, we
extract a fixed-length feature descriptor from several layers (``conv3'',
``conv4'', ``conv5'', and ``context features'').  The descriptors are
L2-normalized, concatenated, re-scaled, and dimension-reduced (1x1 convolution)
to produce a fixed-length feature descriptor for each proposal of size 512x7x7.  Two
fully-connected (FC) layers process each descriptor and produce two outputs: a
one-of-$K$ object class prediction (``softmax''), and an adjustment to the proposal
region's bounding box (``bbox'').

The rest of this section explains the details of ION and motivates why we chose
this particular architecture.



\subsection{Pooling from multiple layers}
\label{sec:pool-multiple}



Recent successful detectors such as Fast R-CNN, Faster
R-CNN~\cite{ren2015faster}, and SPPnet, all pool from the last convolutional
layer (``conv5\_3'') in VGG16~\cite{simonyan2015verydeep}.  In order to extend
this to multiple layers, we must consider issues of dimensionality and
amplitude.

Since we know that pre-training on ImageNet is important to achieve
state-of-the-art performance~\cite{agrawal14analyzing}, and we would like to use
the previously trained VGG16 network~\cite{simonyan2015verydeep}, it is
important to preserve the existing layer shapes.  Therefore, if we want to pool
out of more layers, the final feature must also be shape 512x7x7 so that it is
the correct shape to feed into the first fully-connected layer (fc6).  In
addition to matching the original shape, we must also match the original
activation amplitudes, so that we can feed our feature into fc6.

To match the required 512x7x7 shape, we concatenate each pooled feature along
the channel axis and reduce the dimension with a 1x1 convolution.  To match the
original amplitudes, we L2 normalize each pooled ROI and re-scale back up by an
empirically determined scale.  Our experiments use a ``scale layer'' with a
learnable per-channel scale initialized to 1000 (measured on the training set).
We later show in Section~\ref{sec:norm} that a fixed scale works just as well.



As a final note, as more features are concatenated together, we need to
correspondingly decrease the initial weight magnitudes of the 1x1 convolution,
so we use ``Xavier'' initialization~\cite{glorot10}.

\subsection{Context features with IRNNs}

Our architecture for computing context features in ION is shown in more detail in
Figure~\ref{fig:arch-rnn}.  On top of the last convolutional layer (conv5),
we place RNNs that move \emph{laterally} across the image.  Traditionally, an
RNN moves left-to-right along a sequence, consuming an input at every step,
updating its hidden state, and producing an output.  We extend this to two
dimensions by placing an RNN along each row and along each column of the
image.  We have four RNNs in total that move in the cardinal directions: right,
left, down, up.  The RNNs sit on top of conv5 and produce an output with the
same shape as conv5.

There are many possible forms of recurrent neural networks that we could use:
gated recurrent units (GRU)~\cite{gru}, long short-term memory
(LSTM)~\cite{lstm}, and plain tanh recurrent neural networks.  In this paper, we
explore RNNs composed of rectified linear units (ReLU).  Le \etal~\cite{irnn} recently
showed that these networks are easy to train and are good at modeling long-range
dependencies, if the recurrent weight matrix is initialized to the identity
matrix.  This means that at initialization, gradients are propagated backwards
with full strength.
Le \etal~\cite{irnn} call a ReLU RNN initialized this way an ``IRNN,'' and show
that it performs almost as well as an LSTM for a real-world language modeling
task, and better than an LSTM for a toy memory problem.  We adopt this
architecture because it is very simple to implement and parallelize, and is much
faster than LSTMs or GRUs to compute.

For our problem, we have four independent IRNNs that move in four directions.
To implement the IRNNs as efficiently as possible, we split the internal IRNN
computations into separate logical layers.  Viewed this way, we can see that
the input-to-hidden transition is a 1x1 convolution, and that it can be shared
across different directions.
Sharing this transition allows us to remove 6 conv layers in total with a
negligible effect on accuracy ($-0.1$ mAP).
The bias can be shared in the same way, and
merged into the 1x1 conv layer.  The IRNN layer now only needs to apply
the recurrent matrix and apply the nonlinearity at each step.  The output
from the IRNN is computed by concatenating the hidden state from the four
directions at each spatial location.

This is the update for an IRNN that moves to the right; similar equations exist
for the other directions:
\begin{align}
  h_{i,j}^\text{right} &\gets \max\left(
    \mathbf{W}_{hh}^\text{right} h_{i,j-1}^\text{right} + h_{i,j}^\text{right},
    0 \right).
  \label{eq:rnn}
\end{align}
Notice that the input is not explicitly shown in the equation, and there is no
input-to-hidden transition.  This is because it was computed as part of the 1x1
convolution, and then copied in-place to each hidden layer.
For each direction, we can compute all of the independent rows/columns in
parallel, stepping all IRNNs together with a single matrix multiply.  On a GPU,
this results in large speedups compared to computing each RNN cell one at a
time.

We also explore using semantic segmentation labels to regularize the IRNN
output.  When using these labels, we add the deconvolution and crop layer as
implemented by Long \etal~\cite{fcn}.  The deconvolution upsamples by 16x with a
32x32 kernel, and we add an extra softmax loss layer with a weight of 1.  This
is evaluated in Section~\ref{sec:seg}.

\paragraph{Variants and simplifications.} We explore several
further simplifications.

\begin{packed_enum}
\item We fixed the hidden transition matrix to the identity $\mathbf{W}_{hh}^\text{right}
= I$, which allows us to entirely remove it:
\begin{align}
  h_{i,j}^\text{right} &\gets \max\left( h_{i,j-1}^\text{right} + h_{i,j}^\text{right}, 0 \right).
  \label{eq:no-whh}
\end{align}
This is like an accumulator, but with ReLU after each step.  In
Section~\ref{sec:irnn-arch} we show that removing the recurrent matrix has a
surprisingly small impact.
\item To prevent overfitting, we include dropout layers ($p = 0.25$)
after each concat layer in all experiments.  We later found that in fact the model is
underfitting and there is no need for dropout anywhere in the network.
\item Finally, we trained a separate bias $b_0$ for the first step in the RNN
in each direction. However, since it tends to remain near zero after training,
this component is not really necessary.
\end{packed_enum}

\begin{figure}[t]
  \begin{center}
   \includegraphics[width=0.6\linewidth]{figures/rnn-slice.pdf}
  \end{center}
  \vspace{-6pt}
  \caption{Interpretation of the first IRNN output.  Each cell in the output
  summarizes the features to the left/right/top/bottom.}
  \label{fig:rnn-slice}
  \vspace{-6pt}
\end{figure}

\paragraph{Interpretation.}
After the first 4-directional IRNN (out of the two IRNNs), we obtain a feature map that summarizes
nearby objects at every position in the image.  As illustrated in
Figure~\ref{fig:rnn-slice}, we can see that the first IRNN creates a summary of
the features to the left/right/top/bottom of every cell.  The subsequent 1x1
convolution then mixes this information together as a dimension reduction.

After the second 4-directional IRNN, every cell on the output depends on every
cell of the input.  In this way, our context features are both global and
local.  The features vary by spatial position, and each cell is a global
summary of the image with respect to that specific spatial location.

\section{Conclusion}
This paper introduces the Inside-Outside Net (ION), an architecture that leverages
context and multi-scale knowledge for object detection. Our architecture uses a
2x stacked 4-directional IRNN for context, and multi-layer ROI pooling with
normalization for improved object description.  To justify our design choices,
we conducted extensive experiments evaluating choices like the number of layers
combined, using segmentation loss, normalizing feature amplitudes, different
IRNN architectures, and other variations.  We achieve state-of-the-art results
on both PASCAL VOC and COCO, and find our proposed architecture is particularly
effective at improving detection of small objects.


\section{Design evaluation}

\input{tab_pool_multiple}

In this section, we explore changes to our architecture and justify our design
choices with experiments on PASCAL VOC 2007.  All numbers in this section are
VOC 2007 test mAP, trained on 2007 trainval + 2012 trainval, with the settings
described in Section~\ref{sec:experimental-setup}.  Note that for this section,
we use dropout in all networks, and a single round of bounding box regression at
test time.

\subsection{Pool from which layers?}

As described in Section~\ref{sec:pool-multiple}, our detector pools regions of
interest (ROI) from multiple layers and combines the result.  A straightforward
approach would be to concatenate the ROI from each layer and reduce the
dimensionality using a 1x1 convolution.  As shown in Table
\ref{tab:pool-multiple} (left column), this does not work.
In VGG16, the convolutional features at different layers can have very different
amplitudes, so that naively combining them leads to unstable learning.  While it
is possible in theory to learn a model with inputs of very different
amplitude, this is ill-conditioned and does not work well in practice.
It is necessary to normalize the amplitude such that the features being pooled
from all layers have similar magnitude.  Our method's normalization scheme fixes
this problem, as shown in Table~\ref{tab:pool-multiple} (right column).

\input{tab_norm}

\subsection{How should we normalize feature amplitude?}
\label{sec:norm}

When performing L2 normalization, there are a few choices to be made: do you
sum over channels and perform one normalization per spatial location (as in
ParseNet~\cite{parsenet}), or should you sum over all entries in each pooled ROI
and normalize it as a single blob.  Further, when re-scaling the features back
to an fc6-compatible magnitude, should you use a fixed scale or should you learn a
scale per channel?  The reason why you might want to learn a scale per channel
is that you get more sharing than you would if you relied on the 1x1 convolution
to model the scale.  We evaluate this in Table~\ref{tab:norm}, and find that all
of these approaches perform about the same, and the distinction doesn't matter
for this problem.  The important aspect is whether amplitude is taken into
account; the different schemes we explored in Table~\ref{tab:norm} are all
roughly equivalent in performance.

To determine the initial scale, we measure the mean scale of features pooled
from conv5 on the training set, and use that as the fixed scale.  Using Fast
R-CNN, we measured the mean norm to be approximately 1000 when summing over all
entries, and 130 when summing across channels.

\input{tab_seg}


\subsection{How much does segmentation loss help?}
\label{sec:seg}

Although our target task is object detection, many datasets also have semantic
segmentation labels, where the object class of every pixel is labeled.  Many
images in PASCAL VOC and every image in COCO has these labels.  This is valuable
information that can be incorporated into a training algorithm to improve
performance.

As shown in Figure~\ref{fig:arch-rnn}, when adding stacked IRNNs it is possible
to have them also predict a semantic segmentation output---a multitask setup.
In Table~\ref{tab:seg}, we see that these extra labels consistently provide
about a +1 point boost in mAP for object detection.  This is because we are
training the network with more bits of supervision, so even though we are adding
extra labels that we do not care about during inference, the features inside the
network are trained to contain more information than they would have otherwise
if only trained on object detection.  Since this is an extra layer used only for
training, we can drop the layer at test time and get a +1 mAP point boost with no
change in runtime.

\subsection{How should we incorporate context?}

While RNNs are a powerful mechanism of incorporating context, they are not the only
method.  For example, one could simply add more convolutional layers on top of
conv5 and then pool out of the top convolutional layer.  As shown in
Figure~\ref{fig:global}, stacked 3x3 convolutions add two cells worth of
context, and stacked 5x5 convolutions add 6 cells.  Alternatively, one could
use a global average and unpool (tile or repeat spatially) back to the original
shape as in ParseNet~\cite{parsenet}.

We compared these approaches on VOC 2007 test, shown in Table~\ref{tab:global}.
The 2x stacked 4-dir IRNN layers have fewer parameters than the alternatives, and
perform better on the test set (both with and without segmentation labels).
Therefore, we use this architecture to compute ``context features'' for all
other experiments.

\input{fig_global}

\input{tab_global}

\subsection{Which IRNN architecture?}
\label{sec:irnn-arch}

When designing the IRNN for incorporating context, there are a few basic decisions
to be made, namely how many layers and how many hidden units per layer.
In addition, we explore the idea of entirely removing the recurrent transition
(equivalent to replacing it with the identity matrix), so that
the IRNN consist of repeated steps of: accumulate, ReLU, accumulate, \etc.
Note that this is not the same as an integral/area image, since each step has
ReLU.

As shown in Table~\ref{tab:num-irnn-layers}, using 2 IRNN layers performs the
best on VOC 2007 test.  While stacking more convolution layers tends to make
ConvNets perform better, the same is not always true for
RNNs~\cite{karpathy15rnn}.
We also found that the number of hidden units did not have a strong effect on
the performance (Table~\ref{tab:num-hidden-units}), and chose 512 as the
baseline size for all other experiments.

Finally, we were surprised to discover that removing the recurrent
$\mathbf{W}_{hh}$ transition performs almost as well as learning it
(Table~\ref{tab:num-hidden-units}).  It seems that the input-to-hidden and
hidden-to-output connections contain sufficient context that the
recurrent transition can be removed and replaced with an addition,
saving a large matrix multiply.

\input{tab_num_irnn_layers}

\input{tab_num_hidden_units}

\subsection{Other variations}

There are some other variations on our architecture that perform almost as
well, which we summarize in Table~\ref{tab:other}.  For example, (a) the first
IRNN only processes two directions left/right and the second IRNN only processes
up/down.  This kind of operation was explored in ReNet~\cite{renet} and
performs the same as modeling all four directions in both IRNN layers.  We also
explored (b) pooling out of both IRNNs, and (c) pooling out of both stacked
convolutions and the IRNNs.  None of these variations perform better than our
main method.

\input{tab_other}







\section{Introduction}

Reliably detecting an object requires a variety of information, including the
object's fine-grained details and the context surrounding it.
Current state-of-the-art detection approaches~\cite{MR-CNN,ren2015faster} only
use information near an object's region of interest (ROI).  This places
constraints on the type and accuracy of objects that may be detected.

We explore expanding the approach of~\cite{fast-rcnn} to include two additional
sources of information. The first uses a multi-scale
representation~\cite{koenderink1987jets} that captures fine-grained details by
pooling from multiple lower-level convolutional layers in a ConvNet
\cite{simonyan2015verydeep}. These
skip-layers~\cite{sermanetCVPR13,parsenet,fcn,hariharan2015hypercolumns} span
multiple spatial resolutions and levels of feature abstraction. The information
gained is especially important for small objects, which require the higher
spatial resolution provided by lower-level layers.

Our second addition is the use of contextual information. It is well known in
the study of human and computer vision that context plays an important role in
visual recognition~\cite{torralba2003context,deviExploringTinyImages,Hoiem2009}.
To gather contextual information we explore the use of spatial Recurrent Neural Networks (RNNs).
These RNNs pass spatially varying contextual information both horizontally
and vertically across an image. The use of at least two RNN layers ensures
information may be propagated across the entire image. We compare our approach to other common methods for adding contextual information, including global average pooling and additional convolutional layers.
Global average pooling provides information about the entire image, similar to
the features used for scene or image classification
\cite{GIST,krizhevsky2012imagenet}.


Following previous approaches~\cite{girshick2014rcnn}, we use object proposal
detectors \cite{Hosang15proposals,UijlingsIJCV2013,edgeboxes} to identify ROIs in
an image. Each ROI is then classified as containing one or none of the objects of
interest. Using dynamic pooling~\cite{sppnet} we can efficiently evaluate
thousands of different candidate ROIs with a single forwards pass of the
network. For each candidate ROI, the multi-scale and context information is
concatenated into a single layer and fed through several fully connected layers
for classification.

We demonstrate that both sources of additional information, context and
multi-scale, are complementary in nature. This matches our intuition that
context features look broadly across the image, while multi-scale features
capture more fine-grained details. We show large improvements on the PASCAL
VOC~\cite{pascal} and Microsoft COCO~\cite{coco} object detection datasets and
provide a thorough evaluation of the gains across different object types.  We
find that the they are most significant for object types that have been
historically difficult. For example, we show improved accuracy for potted plants
which are often small and amongst clutter. In general, we find that our
approach is more adept at detecting small objects than previous
state-of-the-art methods.  For heavily occluded objects like chairs, gains are
found when using contextual information.

While the technical methods employed (spatial
RNNs~\cite{nd-rnn,sceneLSTM,renet}, skip-layer
connections~\cite{sermanetCVPR13,parsenet,fcn,hariharan2015hypercolumns}) have
precedents in the literature, we demonstrate that their well-executed
combination has an unexpectedly positive impact on the detector's accuracy.  As
always, the devil is in the details~\cite{Chatfield14} and thus our paper aims
to provide a thorough exploration of design choices and their
outcomes.










\vspace{-3pt}
\paragraph{Contributions.}
We make the following contributions:
\begin{packed_enum}
\item We introduce the ION architecture that leverages context and multi-scale
  skip pooling for object detection.

\item We achieve state-of-the-art results on PASCAL VOC 2007, with a mAP of
  79.2{}\%, VOC 2012, with a mAP of 76.4{}\%,
  and on COCO, with a mAP of 24.9{}\%.

\item We conduct extensive experiments evaluating choices like the number of
layers combined, using a segmentation loss, normalizing feature amplitudes,
different IRNN architectures, and other variations.

\item We analyze the detector's performance and find improved accuracy across
the board, but, in particular, for small objects.
\end{packed_enum}


\section{Lorem ipsum}

\todo{This is here to make pagination work out.}

Lorem ipsum dolor sit amet, consectetur adipiscing elit. Pellentesque eget nulla
risus. Sed sit amet augue risus. Mauris ut tempor lorem, nec bibendum justo.
Donec eu tortor diam. Mauris eget arcu porta, blandit mi sed, tempor mauris.
Fusce vitae fermentum leo. Proin placerat auctor lorem, sed efficitur ante
egestas ut. Ut nec aliquet felis. Quisque varius consectetur dolor, vitae rutrum
quam. Suspendisse id dictum leo, ac egestas mi. Sed sit amet dapibus sapien.
Pellentesque consequat efficitur libero et vestibulum. Aenean ultricies turpis
elit, ut interdum nisi dapibus ac. Quisque sed mollis ex, sed dapibus sapien.
Pellentesque nec tellus sed elit tincidunt tempor. Cras aliquet consectetur
neque.

Integer magna eros, mollis sit amet tellus et, tincidunt molestie turpis.
Praesent maximus tortor lacinia gravida consectetur. Lorem ipsum dolor sit amet,
consectetur adipiscing elit. Sed congue nisl sit amet euismod vestibulum.
Vestibulum ante ipsum primis in faucibus orci luctus et ultrices posuere cubilia
Curae; Vestibulum in nulla vel elit vulputate sagittis. Sed tincidunt diam
ipsum, ac consequat dui tincidunt vel.

Fusce eu sem sit amet felis imperdiet bibendum non quis leo. Duis velit odio,
luctus quis bibendum id, ultricies sit amet enim. In hac habitasse platea
dictumst. Sed arcu tellus, commodo ac congue eget, laoreet eu turpis. Etiam in
est vitae purus rhoncus porta. Aenean luctus fringilla feugiat. Proin sit amet
dapibus felis. Nulla euismod libero ligula, ac rhoncus odio maximus non. Sed
blandit consectetur augue quis tempor. Pellentesque ut libero at metus tempus
laoreet et eget metus. Curabitur euismod justo quis tortor vulputate, eget
viverra quam posuere. Nulla lacinia turpis sollicitudin leo ornare sollicitudin
a sit amet dui. Maecenas lacus erat, tempus rutrum pulvinar sed, pulvinar et
orci. Vivamus ipsum mi, dictum nec enim in, ultricies facilisis justo. Proin ac
augue tortor. Vestibulum fringilla ex et lacus aliquet, sit amet accumsan tortor
ornare.

Cras quis elit volutpat, mollis urna a, lacinia ipsum. Pellentesque suscipit
rhoncus massa vitae vehicula. Nunc sodales nec orci a placerat. Mauris efficitur
sagittis ipsum vitae sollicitudin. Nunc iaculis tincidunt odio. Nunc at eros
scelerisque, tempus justo et, aliquet quam. Duis id erat massa. Fusce id diam ut
lorem rhoncus volutpat. Etiam pharetra sit amet tellus eget accumsan. Nullam
lacinia sit amet justo quis porta. Aliquam dapibus auctor leo, sed pellentesque
tortor interdum a. Fusce rhoncus convallis urna vel sodales. Pellentesque dictum
enim at laoreet pharetra.

Nullam turpis nunc, mattis a finibus eu, tincidunt vitae erat. Pellentesque
congue felis turpis, sed interdum nisl placerat sed. In et metus lobortis odio
luctus finibus eget vitae dui. Fusce et orci ut leo consectetur ornare. Proin
sodales eu nisi vitae interdum. Sed consectetur gravida tincidunt. Nulla in ex
commodo, mattis nisi vel, porta mi. Cras commodo accumsan ligula, eu fermentum
felis molestie ut. Vivamus mauris est, euismod ac lorem nec, pellentesque
sollicitudin augue. Quisque pretium suscipit lorem, vitae consectetur dui
sagittis ut. Morbi lobortis, diam gravida aliquet fringilla, turpis leo
ullamcorper ex, nec dictum lectus velit quis quam. Morbi neque mi, feugiat et
aliquam ut, sollicitudin vel turpis.

Lorem ipsum dolor sit amet, consectetur adipiscing elit. Pellentesque eget nulla
risus. Sed sit amet augue risus. Mauris ut tempor lorem, nec bibendum justo.
Donec eu tortor diam. Mauris eget arcu porta, blandit mi sed, tempor mauris.
Fusce vitae fermentum leo. Proin placerat auctor lorem, sed efficitur ante
egestas ut. Ut nec aliquet felis. Quisque varius consectetur dolor, vitae rutrum
quam. Suspendisse id dictum leo, ac egestas mi. Sed sit amet dapibus sapien.
Pellentesque consequat efficitur libero et vestibulum. Aenean ultricies turpis
elit, ut interdum nisi dapibus ac. Quisque sed mollis ex, sed dapibus sapien.
Pellentesque nec tellus sed elit tincidunt tempor. Cras aliquet consectetur
neque.

Integer magna eros, mollis sit amet tellus et, tincidunt molestie turpis.
Praesent maximus tortor lacinia gravida consectetur. Lorem ipsum dolor sit amet,
consectetur adipiscing elit. Sed congue nisl sit amet euismod vestibulum.
Vestibulum ante ipsum primis in faucibus orci luctus et ultrices posuere cubilia
Curae; Vestibulum in nulla vel elit vulputate sagittis. Sed tincidunt diam
ipsum, ac consequat dui tincidunt vel.

Fusce eu sem sit amet felis imperdiet bibendum non quis leo. Duis velit odio,
luctus quis bibendum id, ultricies sit amet enim. In hac habitasse platea
dictumst. Sed arcu tellus, commodo ac congue eget, laoreet eu turpis. Etiam in
est vitae purus rhoncus porta. Aenean luctus fringilla feugiat. Proin sit amet
dapibus felis. Nulla euismod libero ligula, ac rhoncus odio maximus non. Sed
blandit consectetur augue quis tempor. Pellentesque ut libero at metus tempus
laoreet et eget metus. Curabitur euismod justo quis tortor vulputate, eget
viverra quam posuere. Nulla lacinia turpis sollicitudin leo ornare sollicitudin
a sit amet dui. Maecenas lacus erat, tempus rutrum pulvinar sed, pulvinar et
orci. Vivamus ipsum mi, dictum nec enim in, ultricies facilisis justo. Proin ac
augue tortor. Vestibulum fringilla ex et lacus aliquet, sit amet accumsan tortor
ornare.

Cras quis elit volutpat, mollis urna a, lacinia ipsum. Pellentesque suscipit
rhoncus massa vitae vehicula. Nunc sodales nec orci a placerat. Mauris efficitur
sagittis ipsum vitae sollicitudin. Nunc iaculis tincidunt odio. Nunc at eros
scelerisque, tempus justo et, aliquet quam. Duis id erat massa. Fusce id diam ut
lorem rhoncus volutpat. Etiam pharetra sit amet tellus eget accumsan. Nullam
lacinia sit amet justo quis porta. Aliquam dapibus auctor leo, sed pellentesque
tortor interdum a. Fusce rhoncus convallis urna vel sodales. Pellentesque dictum
enim at laoreet pharetra.

\section{Prior work}

\paragraph{ConvNet object detectors.}
ConvNets with a small number of hidden layers have been used for object
detection for the last two decades (\eg, from~\cite{lecun94} to~\cite{sermanetCVPR13}).  Until
recently, they were successful in restricted domains such as face detection.
Recently, deeper ConvNets have led to radical improvements in the
detection of more general object categories.  This shift came about when the
successful application of deep ConvNets to image
classification~\cite{krizhevsky2012imagenet} was transferred to object detection
in the R-CNN system of Girshick \etal~\cite{girshick2014rcnn} and the OverFeat
system of Sermanet \etal~\cite{overfeat}.  Our work builds on the rapidly
evolving R-CNN (``region-based convolutional neural network'') line of work.
Our experiments are conducted with Fast R-CNN~\cite{fast-rcnn},
which is an end-to-end trainable refinement of He \etal's SPPnet~\cite{sppnet}.
We discuss the relationship of our approach to other methods later in the paper
in the context of our model description and experimental results.

\vspace{-3pt}
\paragraph{Spatial RNNs.}
Recurrent Neural Networks (RNNs) exist in various extended forms, including
bidirectional RNNs~\cite{bi-rnn} that process sequences left-to-right and
right-to-left in parallel.  Beyond simple sequences, RNNs exist in full
multi-dimensional variants, such as those introduced by Graves and
Schmidhuber~\cite{nd-rnn} for handwriting recognition.  As a lower-complexity
alternative, \cite{sceneLSTM,renet} explore running an RNN spatially (or
laterally) over a feature map in place of convolutions. These papers examine
spatial RNNs for the tasks of semantic segmentation and image classification,
respectively.  We employ spatial RNNs as a mechanism for computing contextual
features for use in object detection.

\vspace{-3pt}
\paragraph{Skip-layer connections.}
Skip-layer connections are a classic neural network idea wherein activations
from a lower layer are routed directly to a higher layer while bypassing intermediate layers.  The specifics of the wiring and combination method
differ between models and applications.  Our usage of skip connections is most
closely related to those used by Sermanet \etal~\cite{sermanetCVPR13} (termed
``multi-stage features'') for pedestrian detection.  Different
from~\cite{sermanetCVPR13}, we find it essential to L2 normalize activations
from different layers prior to combining them.

The need for activation normalization when combining features across layers was
recently noted by Liu \etal(ParseNet~\cite{parsenet}) in a model for semantic segmentation
that makes use of global image context features.  Skip connections have also
been popular in recent models for semantic segmentation, such as the ``fully
convolutional networks'' in~\cite{fcn}, and for object instance segmentation,
such as the ``hypercolumn features'' in~\cite{hariharan2015hypercolumns}.


\section{Results}

We train and evaluate our dataset on three major datasets: PASCAL VOC 2007, VOC
2012, and on MS COCO.  We demonstrate state-of-the-art results on all three
datasets.

\subsection{Experimental setup}
\label{sec:experimental-setup}

All of our experiments use Fast R-CNN~\cite{fast-rcnn} built on the
Caffe~\cite{jia2014caffe} framework, and the VGG16
architecture~\cite{simonyan2015verydeep}, all of which are available online.
As is common practice, we use the publicly available weights pre-trained on
ILSVRC2012~\cite{ilsvrc2012} downloaded from the Caffe Model
Zoo.\footnote{\scriptsize{\url{https://github.com/BVLC/caffe/wiki/Model-Zoo}}}

We make some changes to Fast R-CNN, which give a small improvement over the
baseline.  We use 4 images per mini-batch, implemented as 4 forward/backward
passes of single image mini-batches, with gradient accumulation.  We sample 128
ROIs per image leading to 512 ROIs per model update.  We measure the norm of the
parameter gradient vector and rescale it if its L2 norm is above 20 (80 when
accumulating over 4 images).

To accelerate training, we use a two-stage schedule.  As noted by
Girshick~\cite{fast-rcnn}, it is not necessary to fine-tune all layers, and
nearly the same performance can be achieved by fine-tuning starting from
conv3\_1.  With this in mind, we first train for {40k} iterations with
conv1\_1 through conv5\_3 frozen, and then another {100k} iterations with
only conv1\_1 through conv2\_2 frozen.  All other layers are fine-tuned.  When training for COCO, we use 80k
and 320k iterations respectively.  We found that shorter training schedules
are not enough to fully converge.

We also use a different learning rate (LR) schedule.  The LR exponentially
decays from $5 \cdot 10^{-3}$ to $10^{-4}$ in the first stage, and from
$10^{-3}$ to $10^{-5}$ in the second stage.  To reduce the effect of random
variation, we fix the random seed so that all variants see the same images in
the same order.  For PASCAL VOC we use the same pre-computed selective search
boxes from Fast R-CNN, and for COCO we use the boxes precomputed by Hosang
\etal~\cite{Hosang15proposals}.  Finally, we modified the test thresholds in
Fast R-CNN so that we keep only boxes with a softmax score above 0.05, and keep
at most 100 boxes per images.

When re-running the baseline Fast R-CNN using the above settings, we see a +0.8
mAP improvement over the original settings on VOC 2007 test.  We compare
against the baseline using our improved settings where possible.

\input{fig_xs_objects}

\subsection{PASCAL VOC 2007}

As shown in Table~\ref{tab:voc-2007-test}, we evaluate our detector (ION) on PASCAL
VOC 2007, training on the VOC 2007 trainval dataset merged with the 2012
trainval dataset, a common practice.  Applying our method described above, we obtain a mAP of
76.5\%.  We then make some simple modifications, as described below, to achieve
a higher score of 79.2\%.

MR-CNN~\cite{MR-CNN} introduces a bounding box regression scheme to improve
results on VOC, where bounding boxes are evaluated twice: (1) the initial
proposal boxes are evaluated and regressed to improved locations and then (2)
the improved locations are passed again through the network.  All boxes are
accumulated together, and non-max supression is applied.  Finally, a weighted
vote is computed for each kept box (over all boxes, including those
suppressed), where boxes that overlap a kept box by at least 0.5 IoU contribute
to the average.  For our method, we use the softmax scores as the weights.
When adding this scheme to our method, our mAP rises from 76.5\% to 78.5\%.
Finally, we observed that our models are underfitting and we remove dropout from
all layers to get a further gain up to 79.2\%.

MR-CNN also uses context and achieves 78.2\%.  However, we note that their
method requires that pieces are evaluated individually, and thus has a test
runtime around 30 seconds per image, while our method is significantly faster,
taking 0.8s per image on a Titan X GPU (excluding proposal generation) without
two-stage bounding box regression and 1.15s per image with it.

\subsection{PASCAL VOC 2012}

We also evaluate on the slightly more challenging VOC 2012 dataset, submitting
to the public evaluation server.\footnote{Anonymous URL:
\tiny{\url{http://host.robots.ox.ac.uk:8080/anonymous/B3VFLE.html}}}
In Table~\ref{tab:voc-2012-test}, we show the top methods on the public
leaderboard as of the time of submission.  Our detector obtains a mAP of 76.4\%,
which is several points higher than the next best submission, and is the most
accurate for most categories.

\subsection{MS COCO}
\label{sec:results-mscoco}

Microsoft has recently released the Common Objects in Context dataset, which
contains 80k training images (``2014 train'') and 40k validation images (``2014
val'').  There is an associated MS COCO challenge with a new evaluation metric,
that averages mAP over different IoU thresholds, from 0.5 to 0.95 (written as
``0.5:0.95'').  This places a significantly larger emphasis on localization
compared to the PASCAL VOC metric which only requires IoU of 0.5.

We are only aware of one baseline performance number for this dataset, as
published in the Fast R-CNN paper, which cites a mAP of 19.7\% on the
2015 test-dev set~\cite{fast-rcnn}.  We trained our own Fast R-CNN model on
``2014 train'' using our longer training schedule and obtained a higher mAP of
20.5\% mAP on the same set, which we use as a baseline.
As shown in Table~\ref{tab:coco-2015-test}, when trained on the same images with
the same schedule, our method obtains a large improvement over the baseline with
a mAP of 24.9{}\%.

We tried applying the same bounding box voting scheme~\cite{MR-CNN} to COCO, but
found that performance \textit{decreases} on the COCO metric (IOU 0.5:0.95,
second row of Table~\ref{tab:coco-2015-test}).  Interestingly, the scheme increases
performance at IoU 0.5 (the PASCAL metric).  Since the scheme heuristically
blurs together box locations, it can find the general location of objects, but
cannot predict precise box locations, which is important for the new COCO metric.
\newtext{%
  As described in the Appendix, we fixed this for our competition submission by
  raising the voting IoU threshold from 0.5 to $\sim 0.85$.
}

\newtext{%
  We submitted ION to the 2015 MS COCO Detection Challenge and won the Best
  Student Entry with $3^\text{rd}$ place overall.  Using only a single model
  (no ensembling), our submission achieved 31.0\% on test-competition
  score and 31.2\% on test-dev score (Table~\ref{tab:coco-2015-test}).
  After the competition, we further improved our test-dev score to 33.1\% by
  adding left-right flipping and adjusting training parameters.
  See the Appendix for details on our challenge submission.
 
 
 
 
 
}

\subsection{Improvement for small objects}

In general, small objects are challenging for detectors: there are fewer pixels
on the object, they are harder to localize, and there can be many more of them
per image.  Small objects are even more challenging for proposal methods.  For
all experiments, we are using selective search~\cite{UijlingsIJCV2013} for
object proposals, which performs very poorly on small objects in COCO with an
average recall under 10\%~\cite{deepmask}.

We find that our detector shows a large relative improvement in this category.
For COCO, if we look at small\footnote{``Small'' means area $\le 32^2$
px; about 40\% of COCO is ``small.''} objects, average precision and average recall
improve from 4.1\% to \newtext{7.0\%} and from 7.3\% to \newtext{10.7\%} respectively.  We highlight
that this is even higher than the baseline proposal method, which is only
possible because we perform bounding box regression to predict improved box
locations.  Similarly, we show a size breakdown for VOC2007 test in
Figure~\ref{fig:xs-objects} using Hoiem's toolkit for diagnosing
errors~\cite{hoiem12error}, and see similarly large improvements on this dataset as well.


\section*{Visual results on COCO test-dev}

In the directory \texttt{coco-test-dev-detections/}, we show example detections
on COCO 2015 test-dev (first 100 images), using our ION model trained on COCO
2014 train (validation images were not used).  Boxes are annotated with their
softmax score.  All boxes within 50\% of the max score for that image are shown,
so low-scoring boxes may be shown for some images.

\section*{Visual results on VOC 2007 test}

Similarly, \texttt{voc-2007-test-detections/} shows example detections on VOC
2007 test (first 100 images), using our ION model trained on 2007 trainval +
2012 trainval.  Boxes are annotated with their softmax score.  All boxes within
50\% of the max score for that image are shown, so low-scoring boxes may be
shown for some images.

\section*{Additional plots for VOC 2007 error diagnosis}

In this document, we include the full error analysis for ION (our object
detector), tested on PASCAL VOC 2007 test, trained on 2007 trainval + 2012 trainval.
These plots are from Hoiem's toolkit for diagnosing errors~\cite{hoiem12error}.
Figure~{5} in the main paper is a condensed version of these plots, focusing
only on area.

From these plots, we can see that we see large gains across the board, but
particularly for small objects.



{\small
\bibliographystyle{ieee}












