T-CNN: Tubelets with Convolutional Neural Networks for Object Detection from Videos
Michael~Shell
~\IEEEmembership{Member
~IEEE
Kai~Kang*
Hongsheng~Li*
Junjie~Yan
Xingyu~Zeng
Bin~Yang
Tong~Xiao
Cong~Zhang
Zhe~Wang
Ruohui~Wang
Xiaogang~Wang
~\IEEEmembership{Member
~IEEE
Kai~Kang$^{1
....lastname \thanks{...
The state-of-the-art performance for object detection has been significantly improved over the past two years. Besides the introduction of powerful deep neural networks such as GoogleNet \cite{googlenet and VGG \cite{vgg2014simonyan, novel object detection frameworks such as R-CNN \cite{girshick2014rich and its successors, Fast R-CNN \cite{girshick2015fast and Faster R-CNN \cite{ren2015faster, play an essential role in improving the state-of-the-art. Despite their effectiveness on still images, those frameworks are not specifically designed for object detection from videos. Temporal and contextual information of videos are not fully investigated and utilized. In this work, we propose a deep learning framework that incorporates temporal and contextual information from tubelets obtained in videos, which dramatically improves the baseline performance of existing still-image detection frameworks when they are applied to videos. It is called T-CNN, i.e. tubelets with convolutional neueral networks. The proposed framework won newly introduced object-detection-from-video (VID) task with provided data in the ImageNet Large-Scale Visual Recognition Challenge 2015 (ILSVRC 2015). Code is publicly available at \url{https://github.com/myfavouritekk/T-CNN.
csvt_special_issue.tex

%% bare_jrnl_compsoc.tex
%% V1.4b
%% 2015/08/26
%% by Michael Shell
%% See:
%% http://www.michaelshell.org/
%% for current contact information.
%%
%% This is a skeleton file demonstrating the use of IEEEtran.cls
%% (requires IEEEtran.cls version 1.8b or later) with an IEEE
%% Computer Society journal paper.
%%
%% Support sites:
%% http://www.michaelshell.org/tex/ieeetran/
%% http://www.ctan.org/pkg/ieeetran
%% and
%% http://www.ieee.org/

%%*************************************************************************
%% Legal Notice:
%% This code is offered as-is without any warranty either expressed or
%% implied; without even the implied warranty of MERCHANTABILITY or
%% FITNESS FOR A PARTICULAR PURPOSE!
%% User assumes all risk.
%% In no event shall the IEEE or any contributor to this code be liable for
%% any damages or losses, including, but not limited to, incidental,
%% consequential, or any other damages, resulting from the use or misuse
%% of any information contained here.
%%
%% All comments are the opinions of their respective authors and are not
%% necessarily endorsed by the IEEE.
%%
%% This work is distributed under the LaTeX Project Public License (LPPL)
%% ( http://www.latex-project.org/ ) version 1.3, and may be freely used,
%% distributed and modified. A copy of the LPPL, version 1.3, is included
%% in the base LaTeX documentation of all distributions of LaTeX released
%% 2003/12/01 or later.
%% Retain all contribution notices and credits.
%% ** Modified files should be clearly indicated as such, including  **
%% ** renaming them and changing author support contact information. **
%%*************************************************************************


% *** Authors should verify (and, if needed, correct) their LaTeX system  ***
% *** with the testflow diagnostic prior to trusting their LaTeX platform ***
% *** with production work. The IEEE's font choices and paper sizes can   ***
% *** trigger bugs that do not appear when using other class files.       ***                          ***
% The testflow support page is at:
% http://www.michaelshell.org/tex/testflow/


\documentclass[twocolumn]{IEEEtran}
% \documentclass[onecolumn]{IEEEtran}
\usepackage{color}
%
% If IEEEtran.cls has not been installed into the LaTeX system files,
% manually specify the path to it like:
% \documentclass[10pt,journal,compsoc]{../sty/IEEEtran}





% Some very useful LaTeX packages include:
% (uncomment the ones you want to load)


% *** MISC UTILITY PACKAGES ***
%
%\usepackage{ifpdf}
% Heiko Oberdiek's ifpdf.sty is very useful if you need conditional
% compilation based on whether the output is pdf or dvi.
% usage:
% \ifpdf
%   % pdf code
% \else
%   % dvi code
% \fi
% The latest version of ifpdf.sty can be obtained from:
% http://www.ctan.org/pkg/ifpdf
% Also, note that IEEEtran.cls V1.7 and later provides a builtin
% \ifCLASSINFOpdf conditional that works the same way.
% When switching from latex to pdflatex and vice-versa, the compiler may
% have to be run twice to clear warning/error messages.






% *** CITATION PACKAGES ***
%
\ifCLASSOPTIONcompsoc
  % IEEE Computer Society needs nocompress option
  % requires cite.sty v4.0 or later (November 2003)
  \usepackage[nocompress]{cite}
\else
  % normal IEEE
  \usepackage{cite}
\fi
% cite.sty was written by Donald Arseneau
% V1.6 and later of IEEEtran pre-defines the format of the cite.sty package
% \cite{} output to follow that of the IEEE. Loading the cite package will
% result in citation numbers being automatically sorted and properly
% "compressed/ranged". e.g., [1], [9], [2], [7], [5], [6] without using
% cite.sty will become [1], [2], [5]--[7], [9] using cite.sty. cite.sty's
% \cite will automatically add leading space, if needed. Use cite.sty's
% noadjust option (cite.sty V3.8 and later) if you want to turn this off
% such as if a citation ever needs to be enclosed in parenthesis.
% cite.sty is already installed on most LaTeX systems. Be sure and use
% version 5.0 (2009-03-20) and later if using hyperref.sty.
% The latest version can be obtained at:
% http://www.ctan.org/pkg/cite
% The documentation is contained in the cite.sty file itself.
%
% Note that some packages require special options to format as the Computer
% Society requires. In particular, Computer Society  papers do not use
% compressed citation ranges as is done in typical IEEE papers
% (e.g., [1]-[4]). Instead, they list every citation separately in order
% (e.g., [1], [2], [3], [4]). To get the latter we need to load the cite
% package with the nocompress option which is supported by cite.sty v4.0
% and later. Note also the use of a CLASSOPTION conditional provided by
% IEEEtran.cls V1.7 and later.




\usepackage{multirow}
% for checkmarks in tables
\usepackage{pifont}
\newcommand{\cmark}{\ding{51}}%
\newcommand{\xmark}{\ding{55}}%

% *** GRAPHICS RELATED PACKAGES ***
%
\ifCLASSINFOpdf
  \usepackage[pdftex]{graphicx}
  % declare the path(s) where your graphic files are
  \graphicspath{{./figs/}{./final_version/photos/}}
  % and their extensions so you won't have to specify these with
  % every instance of \includegraphics
  \DeclareGraphicsExtensions{.pdf,.jpeg,.jpg,.png}
\else
  % or other class option (dvipsone, dvipdf, if not using dvips). graphicx
  % will default to the driver specified in the system graphics.cfg if no
  % driver is specified.
  % \usepackage[dvips]{graphicx}
  % declare the path(s) where your graphic files are
  % \graphicspath{{../eps/}}
  % and their extensions so you won't have to specify these with
  % every instance of \includegraphics
  % \DeclareGraphicsExtensions{.eps}
\fi
% graphicx was written by David Carlisle and Sebastian Rahtz. It is
% required if you want graphics, photos, etc. graphicx.sty is already
% installed on most LaTeX systems. The latest version and documentation
% can be obtained at:
% http://www.ctan.org/pkg/graphicx
% Another good source of documentation is "Using Imported Graphics in
% LaTeX2e" by Keith Reckdahl which can be found at:
% http://www.ctan.org/pkg/epslatex
%
% latex, and pdflatex in dvi mode, support graphics in encapsulated
% postscript (.eps) format. pdflatex in pdf mode supports graphics
% in .pdf, .jpeg, .png and .mps (metapost) formats. Users should ensure
% that all non-photo figures use a vector format (.eps, .pdf, .mps) and
% not a bitmapped formats (.jpeg, .png). The IEEE frowns on bitmapped formats
% which can result in "jaggedy"/blurry rendering of lines and letters as
% well as large increases in file sizes.
%
% You can find documentation about the pdfTeX application at:
% http://www.tug.org/applications/pdftex






% *** MATH PACKAGES ***
%
%\usepackage{amsmath}
% A popular package from the American Mathematical Society that provides
% many useful and powerful commands for dealing with mathematics.
%
% Note that the amsmath package sets \interdisplaylinepenalty to 10000
% thus preventing page breaks from occurring within multiline equations. Use:
%\interdisplaylinepenalty=2500
% after loading amsmath to restore such page breaks as IEEEtran.cls normally
% does. amsmath.sty is already installed on most LaTeX systems. The latest
% version and documentation can be obtained at:
% http://www.ctan.org/pkg/amsmath





% *** SPECIALIZED LIST PACKAGES ***
%
%\usepackage{algorithmic}
% algorithmic.sty was written by Peter Williams and Rogerio Brito.
% This package provides an algorithmic environment fo describing algorithms.
% You can use the algorithmic environment in-text or within a figure
% environment to provide for a floating algorithm. Do NOT use the algorithm
% floating environment provided by algorithm.sty (by the same authors) or
% algorithm2e.sty (by Christophe Fiorio) as the IEEE does not use dedicated
% algorithm float types and packages that provide these will not provide
% correct IEEE style captions. The latest version and documentation of
% algorithmic.sty can be obtained at:
% http://www.ctan.org/pkg/algorithms
% Also of interest may be the (relatively newer and more customizable)
% algorithmicx.sty package by Szasz Janos:
% http://www.ctan.org/pkg/algorithmicx




% *** ALIGNMENT PACKAGES ***
%
%\usepackage{array}
% Frank Mittelbach's and David Carlisle's array.sty patches and improves
% the standard LaTeX2e array and tabular environments to provide better
% appearance and additional user controls. As the default LaTeX2e table
% generation code is lacking to the point of almost being broken with
% respect to the quality of the end results, all users are strongly
% advised to use an enhanced (at the very least that provided by array.sty)
% set of table tools. array.sty is already installed on most systems. The
% latest version and documentation can be obtained at:
% http://www.ctan.org/pkg/array


% IEEEtran contains the IEEEeqnarray family of commands that can be used to
% generate multiline equations as well as matrices, tables, etc., of high
% quality.




% *** SUBFIGURE PACKAGES ***
%\ifCLASSOPTIONcompsoc
%  \usepackage[caption=false,font=footnotesize,labelfont=sf,textfont=sf]{subfig}
%\else
%  \usepackage[caption=false,font=footnotesize]{subfig}
%\fi
% subfig.sty, written by Steven Douglas Cochran, is the modern replacement
% for subfigure.sty, the latter of which is no longer maintained and is
% incompatible with some LaTeX packages including fixltx2e. However,
% subfig.sty requires and automatically loads Axel Sommerfeldt's caption.sty
% which will override IEEEtran.cls' handling of captions and this will result
% in non-IEEE style figure/table captions. To prevent this problem, be sure
% and invoke subfig.sty's "caption=false" package option (available since
% subfig.sty version 1.3, 2005/06/28) as this is will preserve IEEEtran.cls
% handling of captions.
% Note that the Computer Society format requires a sans serif font rather
% than the serif font used in traditional IEEE formatting and thus the need
% to invoke different subfig.sty package options depending on whether
% compsoc mode has been enabled.
%
% The latest version and documentation of subfig.sty can be obtained at:
% http://www.ctan.org/pkg/subfig




% *** FLOAT PACKAGES ***
%
%\usepackage{fixltx2e}
% fixltx2e, the successor to the earlier fix2col.sty, was written by
% Frank Mittelbach and David Carlisle. This package corrects a few problems
% in the LaTeX2e kernel, the most notable of which is that in current
% LaTeX2e releases, the ordering of single and double column floats is not
% guaranteed to be preserved. Thus, an unpatched LaTeX2e can allow a
% single column figure to be placed prior to an earlier double column
% figure.
% Be aware that LaTeX2e kernels dated 2015 and later have fixltx2e.sty's
% corrections already built into the system in which case a warning will
% be issued if an attempt is made to load fixltx2e.sty as it is no longer
% needed.
% The latest version and documentation can be found at:
% http://www.ctan.org/pkg/fixltx2e


%\usepackage{stfloats}
% stfloats.sty was written by Sigitas Tolusis. This package gives LaTeX2e
% the ability to do double column floats at the bottom of the page as well
% as the top. (e.g., "\begin{figure*}[!b]" is not normally possible in
% LaTeX2e). It also provides a command:
%\fnbelowfloat
% to enable the placement of footnotes below bottom floats (the standard
% LaTeX2e kernel puts them above bottom floats). This is an invasive package
% which rewrites many portions of the LaTeX2e float routines. It may not work
% with other packages that modify the LaTeX2e float routines. The latest
% version and documentation can be obtained at:
% http://www.ctan.org/pkg/stfloats
% Do not use the stfloats baselinefloat ability as the IEEE does not allow
% \baselineskip to stretch. Authors submitting work to the IEEE should note
% that the IEEE rarely uses double column equations and that authors should try
% to avoid such use. Do not be tempted to use the cuted.sty or midfloat.sty
% packages (also by Sigitas Tolusis) as the IEEE does not format its papers in
% such ways.
% Do not attempt to use stfloats with fixltx2e as they are incompatible.
% Instead, use Morten Hogholm'a dblfloatfix which combines the features
% of both fixltx2e and stfloats:
%
% \usepackage{dblfloatfix}
% The latest version can be found at:
% http://www.ctan.org/pkg/dblfloatfix




%\ifCLASSOPTIONcaptionsoff
%  \usepackage[nomarkers]{endfloat}
% \let\MYoriglatexcaption\caption
% \renewcommand{\caption}[2][\relax]{\MYoriglatexcaption[#2]{#2}}
%\fi
% endfloat.sty was written by James Darrell McCauley, Jeff Goldberg and
% Axel Sommerfeldt. This package may be useful when used in conjunction with
% IEEEtran.cls'  captionsoff option. Some IEEE journals/societies require that
% submissions have lists of figures/tables at the end of the paper and that
% figures/tables without any captions are placed on a page by themselves at
% the end of the document. If needed, the draftcls IEEEtran class option or
% \CLASSINPUTbaselinestretch interface can be used to increase the line
% spacing as well. Be sure and use the nomarkers option of endfloat to
% prevent endfloat from "marking" where the figures would have been placed
% in the text. The two hack lines of code above are a slight modification of
% that suggested by in the endfloat docs (section 8.4.1) to ensure that
% the full captions always appear in the list of figures/tables - even if
% the user used the short optional argument of \caption[]{}.
% IEEE papers do not typically make use of \caption[]'s optional argument,
% so this should not be an issue. A similar trick can be used to disable
% captions of packages such as subfig.sty that lack options to turn off
% the subcaptions:
% For subfig.sty:
% \let\MYorigsubfloat\subfloat
% \renewcommand{\subfloat}[2][\relax]{\MYorigsubfloat[]{#2}}
% However, the above trick will not work if both optional arguments of
% the \subfloat command are used. Furthermore, there needs to be a
% description of each subfigure *somewhere* and endfloat does not add
% subfigure captions to its list of figures. Thus, the best approach is to
% avoid the use of subfigure captions (many IEEE journals avoid them anyway)
% and instead reference/explain all the subfigures within the main caption.
% The latest version of endfloat.sty and its documentation can obtained at:
% http://www.ctan.org/pkg/endfloat
%
% The IEEEtran \ifCLASSOPTIONcaptionsoff conditional can also be used
% later in the document, say, to conditionally put the References on a
% page by themselves.




% *** PDF, URL AND HYPERLINK PACKAGES ***
%
\usepackage{url}
% url.sty was written by Donald Arseneau. It provides better support for
% handling and breaking URLs. url.sty is already installed on most LaTeX
% systems. The latest version and documentation can be obtained at:
% http://www.ctan.org/pkg/url
% Basically, \url{my_url_here}.





% *** Do not adjust lengths that control margins, column widths, etc. ***
% *** Do not use packages that alter fonts (such as pslatex).         ***
% There should be no need to do such things with IEEEtran.cls V1.6 and later.
% (Unless specifically asked to do so by the journal or conference you plan
% to submit to, of course. )


% correct bad hyphenation here
\hyphenation{op-tical net-works semi-conduc-tor}


\begin{document}
%
% paper title
% Titles are generally capitalized except for words such as a, an, and, as,
% at, but, by, for, in, nor, of, on, or, the, to and up, which are usually
% not capitalized unless they are the first or last word of the title.
% Linebreaks \\ can be used within to get better formatting as desired.
% Do not put math or special symbols in the title.
\title{T-CNN: Tubelets with Convolutional Neural Networks for Object Detection from Videos}
%
%
% author names and IEEE memberships
% note positions of commas and nonbreaking spaces ( ~ ) LaTeX will not break
% a structure at a ~ so this keeps an author's name from being broken across
% two lines.
% use \thanks{} to gain access to the first footnote area
% a separate \thanks must be used for each paragraph as LaTeX2e's \thanks
% was not built to handle multiple paragraphs
%
%
%\IEEEcompsocitemizethanks is a special \thanks that produces the bulleted
% lists the Computer Society journals use for "first footnote" author
% affiliations. Use \IEEEcompsocthanksitem which works much like \item
% for each affiliation group. When not in compsoc mode,
% \IEEEcompsocitemizethanks becomes like \thanks and
% \IEEEcompsocthanksitem becomes a line break with idention. This
% facilitates dual compilation, although admittedly the differences in the
% desired content of \author between the different types of papers makes a
% one-size-fits-all approach a daunting prospect. For instance, compsoc
% journal papers have the author affiliations above the "Manuscript
% received ..."  text while in non-compsoc journals this is reversed. Sigh.

% \author{Michael~Shell,~\IEEEmembership{Member,~IEEE,}
%         John~Doe,~\IEEEmembership{Fellow,~OSA,}
%         and~Jane~Doe,~\IEEEmembership{Life~Fellow,~IEEE}% <-this % stops a space
% \IEEEcompsocitemizethanks{\IEEEcompsocthanksitem M. Shell was with the Department
% of Electrical and Computer Engineering, Georgia Institute of Technology, Atlanta,
% GA, 30332.\protect\\
% % note need leading \protect in front of \\ to get a newline within \thanks as
% % \\ is fragile and will error, could use \hfil\break instead.
% E-mail: see http://www.michaelshell.org/contact.html
% \IEEEcompsocthanksitem J. Doe and J. Doe are with Anonymous University.}% <-this % stops an unwanted space
% \thanks{Manuscript received April 19, 2005; revised August 26, 2015.}
% }
\author{Kai~Kang*, Hongsheng~Li*, Junjie~Yan, Xingyu~Zeng, Bin~Yang, Tong~Xiao, Cong~Zhang, Zhe~Wang, Ruohui~Wang, Xiaogang~Wang,~\IEEEmembership{Member,~IEEE,}, and Wanli~Ouyang,~\IEEEmembership{Senior~Member,~IEEE,}
\thanks{Copyright © 2017 IEEE. Personal use of this material is permitted. However, permission to use this material for any other purposes must be obtained from the IEEE by sending an email to pubs-permissions@ieee.org.}
\thanks{This work is supported in part by SenseTime Group Limited, in part by the General Research Fund through the Research Grants Council of Hong Kong under Grants CUHK14213616, CUHK14206114, CUHK14205615, CUHK419412, CUHK14203015, CUHK14239816, CUHK14207814, in part by the Hong Kong Innovation and Technology Support Programme Grant ITS/121/15FX, in part by the China Postdoctoral Science Foundation under Grant 2014M552339, in part by National Natural Science Foundation of China (No. 61371192), and in part by ONR N00014-15-1-2356.}
\thanks{*Kai Kang and Hongsheng Li share co-first authorship.}
\thanks{Wanli Ouyang is the corresponding author. (wlouyang@ee.cuhk.edu.hk)}
\thanks{Kai Kang, Hongsheng Li, Tong Xiao, Zhe Wang, Ruohui Wang, Xiaogang Wang, and Wanli Ouyang are with The Chinese University of Hong Kong.}
\thanks{Cong Zhang is with Shanghai Jiao Tong University, China.}
\thanks{Junjie Yan, Xingyu Zeng are with the SenseTime Group Limited.}
\thanks{Bin Yang is with the Computer Science Department, University of Toronto.}

}% <-this % stops an unwanted space
% \author{Kai~Kang$^{1}$\thanks{Co-first authors},~ Hongsheng Li$^{1}$\footnotemark[1],~ Junjie Yan$^{2}$,~ Xingyu Zeng$^{1}$,~ Bin Yang$^{2}$,~ Tong Xiao$^{1}$\\
% Cong Zhang$^{1}$,~ Zhe Wang$^{1}$,~ Ruohui Wang$^{1}$,~ Xiaogang Wang$^{1}$,~ Wanli Ouyang$^{1}$\thanks{Corresponding author. Email: {\tt\small wlouyang@ee.cuhk.edu.hk} }\\
% \small $^{1}$The Chinese University of Hong Kong\\
% \small $^{2}$SenseTime Group\\
% }


% note the % following the last \IEEEmembership and also \thanks -
% these prevent an unwanted space from occurring between the last author name
% and the end of the author line. i.e., if you had this:
%
% \author{....lastname \thanks{...} \thanks{...} }
%                     ^------------^------------^----Do not want these spaces!
%
% a space would be appended to the last name and could cause every name on that
% line to be shifted left slightly. This is one of those "LaTeX things". For
% instance, "\textbf{A} \textbf{B}" will typeset as "A B" not "AB". To get
% "AB" then you have to do: "\textbf{A}\textbf{B}"
% \thanks is no different in this regard, so shield the last } of each \thanks
% that ends a line with a % and do not let a space in before the next \thanks.
% Spaces after \IEEEmembership other than the last one are OK (and needed) as
% you are supposed to have spaces between the names. For what it is worth,
% this is a minor point as most people would not even notice if the said evil
% space somehow managed to creep in.



% The paper headers
% \markboth{Journal of \LaTeX\ Class Files,~Vol.~14, No.~8, August~2015}%
% {Shell \MakeLowercase{\textit{et al.}}: Bare Demo of IEEEtran.cls for Computer Society Journals}
% The only time the second header will appear is for the odd numbered pages
% after the title page when using the twoside option.
%
% *** Note that you probably will NOT want to include the author's ***
% *** name in the headers of peer review papers.                   ***
% You can use \ifCLASSOPTIONpeerreview for conditional compilation here if
% you desire.



% The publisher's ID mark at the bottom of the page is less important with
% Computer Society journal papers as those publications place the marks
% outside of the main text columns and, therefore, unlike regular IEEE
% journals, the available text space is not reduced by their presence.
% If you want to put a publisher's ID mark on the page you can do it like
% this:
%\IEEEpubid{0000--0000/00\$00.00~\copyright~2015 IEEE}
% or like this to get the Computer Society new two part style.
%\IEEEpubid{\makebox[\columnwidth]{\hfill 0000--0000/00/\$00.00~\copyright~2015 IEEE}%
%\hspace{\columnsep}\makebox[\columnwidth]{Published by the IEEE Computer Society\hfill}}
% Remember, if you use this you must call \IEEEpubidadjcol in the second
% column for its text to clear the IEEEpubid mark (Computer Society jorunal
% papers don't need this extra clearance.)



% use for special paper notices
%\IEEEspecialpapernotice{(Invited Paper)}



% for Computer Society papers, we must declare the abstract and index terms
% PRIOR to the title within the \IEEEtitleabstractindextext IEEEtran
% command as these need to go into the title area created by \maketitle.
% As a general rule, do not put math, special symbols or citations
% in the abstract or keywords.
\IEEEtitleabstractindextext{%
\begin{abstract}
The state-of-the-art performance for object detection has been significantly improved over the past two years.
Besides the introduction of powerful deep neural networks such as GoogleNet \cite{googlenet} and VGG \cite{vgg2014simonyan}, novel object detection frameworks such as R-CNN \cite{girshick2014rich} and its successors, Fast R-CNN \cite{girshick2015fast} and Faster R-CNN \cite{ren2015faster}, play an essential role in improving the state-of-the-art.
Despite their effectiveness on still images, those frameworks are not specifically designed for object detection from videos.
Temporal and contextual information of videos are not fully investigated and utilized.
In this work, we propose a deep learning framework that incorporates temporal and contextual information from tubelets obtained in videos, which dramatically improves the baseline performance of existing still-image detection frameworks when they are applied to videos. It is called T-CNN, i.e. tubelets with convolutional neueral networks. The proposed framework won newly introduced object-detection-from-video (VID) task with provided data in the ImageNet Large-Scale Visual Recognition Challenge 2015 (ILSVRC 2015). Code is publicly available at \url{https://github.com/myfavouritekk/T-CNN}.

\end{abstract}

% Note that keywords are not normally used for peerreview papers.
% \begin{IEEEkeywords}
% object detection, tubelets, deep learning, ImageNet.
% \end{IEEEkeywords}
} % abstract


% make the title area
\maketitle


% To allow for easy dual compilation without having to reenter the
% abstract/keywords data, the \IEEEtitleabstractindextext text will
% not be used in maketitle, but will appear (i.e., to be "transported")
% here as \IEEEdisplaynontitleabstractindextext when the compsoc
% or transmag modes are not selected <OR> if conference mode is selected
% - because all conference papers position the abstract like regular
% papers do.
\IEEEdisplaynontitleabstractindextext
% \IEEEdisplaynontitleabstractindextext has no effect when using
% compsoc or transmag under a non-conference mode.



% For peer review papers, you can put extra information on the cover
% page as needed:
% \ifCLASSOPTIONpeerreview
% \begin{center} \bfseries EDICS Category: 3-BBND \end{center}
% \fi
%
% For peerreview papers, this IEEEtran command inserts a page break and
% creates the second title. It will be ignored for other modes.
\IEEEpeerreviewmaketitle



% \IEEEraisesectionheading{\section{Introduction}\label{sec:introduction}}
% Computer Society journal (but not conference!) papers do something unusual
% with the very first section heading (almost always called "Introduction").
% They place it ABOVE the main text! IEEEtran.cls does not automatically do
% this for you, but you can achieve this effect with the provided
% \IEEEraisesectionheading{} command. Note the need to keep any \label that
% is to refer to the section immediately after \section in the above as
% \IEEEraisesectionheading puts \section within a raised box.




% The very first letter is a 2 line initial drop letter followed
% by the rest of the first word in caps (small caps for compsoc).
%
% form to use if the first word consists of a single letter:
% \IEEEPARstart{A}{demo} file is ....
%
% form to use if you need the single drop letter followed by
% normal text (unknown if ever used by the IEEE):
% \IEEEPARstart{A}{}demo file is ....
%
% Some journals put the first two words in caps:
% \IEEEPARstart{T}{his demo} file is ....
%
% Here we have the typical use of a "T" for an initial drop letter
% and "HIS" in caps to complete the first word.
% \IEEEPARstart{T}{his} demo file is intended to serve as a ``starter file''
% for IEEE Computer Society journal papers produced under \LaTeX\ using
% IEEEtran.cls version 1.8b and later.
% % You must have at least 2 lines in the paragraph with the drop letter
% % (should never be an issue)
% I wish you the best of success.

% \hfill mds

% \hfill August 26, 2015

% \subsection{Subsection Heading Here}
% Subsection text here.

% % needed in second column of first page if using \IEEEpubid
% %\IEEEpubidadjcol

% \subsubsection{Subsubsection Heading Here}
% Subsubsection text here.


% An example of a floating figure using the graphicx package.
% Note that \label must occur AFTER (or within) \caption.
% For figures, \caption should occur after the \includegraphics.
% Note that IEEEtran v1.7 and later has special internal code that
% is designed to preserve the operation of \label within \caption
% even when the captionsoff option is in effect. However, because
% of issues like this, it may be the safest practice to put all your
% \label just after \caption rather than within \caption{}.
%
% Reminder: the "draftcls" or "draftclsnofoot", not "draft", class
% option should be used if it is desired that the figures are to be
% displayed while in draft mode.
%
%\begin{figure}[!t]
%\centering
%\includegraphics[width=2.5in]{myfigure}
% where an .eps filename suffix will be assumed under latex,
% and a .pdf suffix will be assumed for pdflatex; or what has been declared
% via \DeclareGraphicsExtensions.
%\caption{Simulation results for the network.}
%\label{fig_sim}
%\end{figure}

% Note that the IEEE typically puts floats only at the top, even when this
% results in a large percentage of a column being occupied by floats.
% However, the Computer Society has been known to put floats at the bottom.


% An example of a double column floating figure using two subfigures.
% (The subfig.sty package must be loaded for this to work.)
% The subfigure \label commands are set within each subfloat command,
% and the \label for the overall figure must come after \caption.
% \hfil is used as a separator to get equal spacing.
% Watch out that the combined width of all the subfigures on a
% line do not exceed the text width or a line break will occur.
%
%\begin{figure*}[!t]
%\centering
%\subfloat[Case I]{\includegraphics[width=2.5in]{box}%
%\label{fig_first_case}}
%\hfil
%\subfloat[Case II]{\includegraphics[width=2.5in]{box}%
%\label{fig_second_case}}
%\caption{Simulation results for the network.}
%\label{fig_sim}
%\end{figure*}
%
% Note that often IEEE papers with subfigures do not employ subfigure
% captions (using the optional argument to \subfloat[]), but instead will
% reference/describe all of them (a), (b), etc., within the main caption.
% Be aware that for subfig.sty to generate the (a), (b), etc., subfigure
% labels, the optional argument to \subfloat must be present. If a
% subcaption is not desired, just leave its contents blank,
% e.g., \subfloat[].


% An example of a floating table. Note that, for IEEE style tables, the
% \caption command should come BEFORE the table and, given that table
% captions serve much like titles, are usually capitalized except for words
% such as a, an, and, as, at, but, by, for, in, nor, of, on, or, the, to
% and up, which are usually not capitalized unless they are the first or
% last word of the caption. Table text will default to \footnotesize as
% the IEEE normally uses this smaller font for tables.
% The \label must come after \caption as always.
%
%\begin{table}[!t]
%% increase table row spacing, adjust to taste
%\renewcommand{\arraystretch}{1.3}
% if using array.sty, it might be a good idea to tweak the value of
% \extrarowheight as needed to properly center the text within the cells
%\caption{An Example of a Table}
%\label{table_example}
%\centering
%% Some packages, such as MDW tools, offer better commands for making tables
%% than the plain LaTeX2e tabular which is used here.
%\begin{tabular}{|c||c|}
%\hline
%One & Two\\
%\hline
%Three & Four\\
%\hline
%\end{tabular}
%\end{table}


% Note that the IEEE does not put floats in the very first column
% - or typically anywhere on the first page for that matter. Also,
% in-text middle ("here") positioning is typically not used, but it
% is allowed and encouraged for Computer Society conferences (but
% not Computer Society journals). Most IEEE journals/conferences use
% top floats exclusively.
% Note that, LaTeX2e, unlike IEEE journals/conferences, places
% footnotes above bottom floats. This can be corrected via the
% \fnbelowfloat command of the stfloats package.



% \vspace{0.2cm}
\section{Introduction}
\label{sec:introduction}
% improvements in object detection
\IEEEPARstart{I}{n} the last several years, the performance of object detection has been significantly improved with the success of novel deep convolutional neural networks (CNN) \cite{he2016deep,googlenet,vgg2014simonyan,ioffe2015batch} and object detection frameworks \cite{girshick2014rich,ouyang2015deepid,girshick2015fast,ren2015faster}.
% frameworks and summary
The state-of-the-art frameworks for object detection such as R-CNN \cite{girshick2014rich} and its successors \cite{girshick2015fast,ren2015faster} extract deep convolutional features from region proposals and classify the proposals into different classes.
DeepID-Net \cite{ouyang2015deepid} improved R-CNN by introducing box pre-training, cascading on region proposals, deformation layers and context representations.
% VID task
Recently, ImageNet introduces a new challenge for object detection from videos (VID), which brings object detection into the video domain.
In this challenge, an object detection system is required to automatically annotate every object in $30$ classes with its bounding box and class label in each frame of the videos, while test videos have no extra information pre-assigned, such as user tags. VID has a broad range of applications on video analysis.

\begin{figure}[tb]
    \centering
    \includegraphics[width=\linewidth]{limitations}
    % \fbox{\rule{0pt}{2in} \rule{0.9\linewidth}{0pt}}
    \caption{Limitations of still-image detectors on videos. (a) Detections from still-image detectors contain large temporal fluctuations, because they do not incorporate temporal consistency and constraints. (b) Still-image detectors may generate false positives solely based the information on single frames, while these false positives can be distinguished considering the context information of the whole video.}
    \label{fig:motivation}
\end{figure}

% drawbacks - temporal, contextual
Despite their effectiveness on still images, still-image object detection frameworks are not specifically designed for videos.
One key element of videos is temporal information, because locations and appearances of objects in videos should be temporally consistent, i.e. the detection results should not have dramatic changes over time in terms of both bounding box locations and detection confidences.
However, if still-image object detection frameworks are directly applied to videos, the detection confidences of an object show dramatic changes between adjacent frames and large long-term temporal variations, as shown by an example in Fig.~\ref{fig:motivation} (a).

One intuition to improve temporal consistency is to propagate detection results to neighbor frames to reduce sudden changes of detection results.
If an object exists in a certain frame, the adjacent frames are likely to contain the same object at neighboring locations with similar confidence.
In other words, detection results can be propagated to adjacent frames according to motion information so as to reduce missed detections. The resulted duplicate boxes can be easily removed by non-maximum suppression (NMS).

Another intuition to improve temporal consistency is to impose long-term constraints on the detection results.
As shown in Fig.~\ref{fig:motivation} (a), the detection scores of a sequence of bounding boxes of an object have large fluctuations over time.
These box sequences, or tubelets, can be generated by tracking and spatio-temporal object proposal algorithms \cite{Oneata:2014spatio}.
A tubelet can be treated as a unit to apply the long-term constraint.
Low detection confidence on some positive bounding boxes may result from moving blur, bad poses, or lack of enough training samples under particular poses.
Therefore, if most bounding boxes of a tubelet have high confidence detection scores, the low confidence scores at certain frames should be increased to enforce its long-term consistency.

Besides temporal information, contextual information is also a key element of videos compared with still images.
Although image context information has been investigated \cite{ouyang2015deepid} and incorporated into still-image detection frameworks, a video, as a collection of hundreds of images, has much richer contextual information.
As shown in Fig.~\ref{fig:motivation} (b), a small amount of frames in a video may have high confidence false positives on some background objects.
Contextual information within a single frame is sometimes not enough to distinguish these false positives.
% However, considering the majority of high-confidence detection results within a video clip, global high-confidence classes in this video can be inferred and used to suppress the confidences of those false positives of other classes.
However, considering the majority of high-confidence detection results within a video clip, the false positives can be treated as outliers and then their detection confidences can be suppressed.
% solutions


The contribution of this works is three-folded.
1) We propose a deep learning framework that extends popular still-image detection frameworks (R-CNN and Faster R-CNN) to solve the problem of general object detection in videos by incorporating temporal and contextual information from tubelets. It is called T-CNN, i.e. tubelets with convolution neural network.
2) Temporal information is effectively incorporated into the proposed detection framework by locally propagating detection results across adjacent frames as well as globally revising detection confidences along tubelets generated from tracking algorithms.
3) Contextual information is utilized to suppress detection scores of low-confidence classes based on all detection results within a video clip.
This framework is responsible for winning the VID task with provided data and achieving the second place with external data in ImageNet Large-Scale Visual Recognition Challenge 2015 (ILSVRC2015).
Code is available at \url{https://github.com/myfavouritekk/T-CNN}.


% section introduction (end)


\section{Related Work} % (fold)
\label{sec:related_work}
\textbf{Object detection from still images.}
State-of-the-art methods for detecting objects of general classes are mainly based on deep CNNs \cite{googlenet,sermanet2013overfeat,girshick2014rich,girshick2015fast,ren2015faster,erhan2014scalable,ouyang2015deepid,redmon2016you,gidaris2015object,girshick2015deformable,mrowca2015spatial,bell2015inside,yan2015object,he2016deep,wang2016dictionary}. Girshick et al. \cite{girshick2014rich} proposed a multi-stage pipeline called Regions with Convolutional Neural Networks
(R-CNN) for training deep CNNs to classify region proposals for object detection. It decomposed the detection problem into several stages including bounding-box proposal, CNN pre-training, CNN fine-tuning, SVM training, and bounding box regression. Such framework showed good performance and was widely adopted in other works. Szegedy et al. \cite{googlenet} proposed the GoogLeNet with a 22-layer structure and ``inception'' modules to replace the CNN in the R-CNN, which won the ILSVRC 2014 object detection task. Ouyang et al. \cite{ouyang2015deepid} proposed a deformation constrained pooling layer and a box pre-training strategy, which achieved an accuracy of $50.3\%$ on the ILSVRC 2014 test set. To accelerate the training of the R-CNN pipeline, Fast R-CNN \cite{girshick2015fast} was proposed, where each image patch was no longer wrapped to a fixed size before being fed into CNN. Instead, the corresponding features were cropped from the output feature maps of the last convolutional layer. In the Faster R-CNN pipeline \cite{ren2015faster}, the region proposals were generated by a Region Proposal Network (RPN), and the overall framework can thus be trained in an end-to-end manner. {He et al. proposed a novel Residual Neural Network (ResNet) \cite{he2016deep} based on residual blocks, which enables training very deep networks with over one hundred layers. Based on the Faster R-CNN framework, He et al. utilized ResNet to win the detection challenges in ImageNet 2015 and COCO 2015. The ResNet has later been applied to many other tasks and proven its effectiveness. Besides region based frameworks, some direct regression frameworks have also been proposed for object detection. YOLO \cite{redmon2016you} divided the image into even grids and simultaneously predicted the bounding boxes and classification scores. SSD \cite{liu2016ssd} generated multiple anchor boxes for each feature map location so as to predict bounding box regression and classification scores for bounding boxes with different scales and aspect ratios.} All these pipelines were for object detection from still images. When they are directly applied to videos in a frame-by-frame manner, they might miss some positive samples because objects might not be of their best poses at certain frames of videos.

{
\textbf{Object detection in videos.}
Since the introduction of ImageNet VID dataset in 2015, there has been multiple works that solve the video object detection problem. 
Han et al. \cite{han2016seq} proposed a sequence NMS method to associated still-image detections into sequences and apply the sequence-level NMS on the results. Weaker class scores are boosted by the detection on the same sequence. Galteri et al. \cite{galteri2017spatio} proposed a closed-loop framework to use object detection results on the previous frame to feed back to the proposal algorithm to improve window ranking. Kang et al. \cite{kang2017object} proposed a tubelet proposal network to efficiently generates hundreds of tubelet proposals simultaneously. 
}

% \vspace{0.1cm}\noindent \textbf{Single-class detection in videos.}
% Apart from the still-image object detection, object detection has also been studied in the video domain. These works, however, usually focus on specific classes such as pedestrians \cite{andriluka2008people}, vehicles [] and faces [], rather than general multiple classes.
% \cite{andriluka2008people} combines


\textbf{Object localization in videos.}
There have also been works on object localization and co-localization~\cite{Prest:2012learning,Papazoglou:2013fast,Joulin:2014efficient,Kwak:2015unsupervised,Maxime:2015is}.
Although such a task seems to be similar, the VID task we focus on is much more challenging.
There are crucial differences between the two problems.
1) The (co)localization problem only requires localizing one of the ground-truth objects of a known (weakly supervised setting) or unknown (unsupervised setting) class in each test frame. In contrast, the VID requires annotating every object from all target classes in each frame.
2) The CorLoc metric is used for localization problem, which is the percentage of test frames in which one of the ground-truth objects is correctly localized with intersection-over-union (IOU) $>0.5$. In VID, the mean average precision (Mean AP) is used for evaluation, which firstly calculates average precisions over all recall rates for each target class and then takes the mean value of all the average precisions.
3) The object (co)localization datasets \cite{Prest:2012learning,rubinstein2013unsupervised} contain annotations on only a few frames (\textit{e.g.}, $1$ frame for each training clip in the YouTubeObjects dataset), while VID dataset contains annotations for different classes on every training and test frames.
The much richer annotations not only enable investigating supervised deep learning methods which were not possible before, but also evaluates algorithm performances more precisely.
With the differences above, the VID task is much more challenging and promising for real-world applications. The previous works on object localization cannot be directly applied to the VID task, while the VID framework can be reduced to solve the object localization problem.

\textbf{Image classification.} The performance of image classification has been significantly improved during the past few years thanks to the large scale datasets \cite{deng2009imagenet} as well as novel deep neural networks and methods \cite{krizhevsky2012imagenet,googlenet,vgg2014simonyan,he2016deep,wang2016cost}. The models for object detection are commonly pre-trained on the ImageNet $1000$-class classification task. Batch normalization layer was proposed in \cite{ioffe2015batch} to reduce the statistical variations among mini batches and accelerate the training process. Simonyan et al. proposed a $19$-layer neural network with very small $3\times3$ convolution kernels in \cite{vgg2014simonyan}, which was proved effective in other related tasks such as detection \cite{girshick2015fast,ren2015faster}, action recognition \cite{Simonyan:2014twostream}, and semantic segmentation \cite{long2015fully}.


\textbf{Visual tracking.}
Object tracking has been studied for decades \cite{possegger2014occlusion,li2015reliable,hong2015multi,maggio2007adaptive,maggio2008efficient,cavallaro2005tracking,yang2009robust,li2017learning}. Recently, deep CNNs have been used for object tracking and achieved impressive tracking accuracy. Wang et al. \cite{wang2015visual} proposed to create an object-specific tracker by online selecting the most influential features from an ImageNet pre-trained CNN, which outperforms state-of-the-art trackers by a large margin.
Nam et al. \cite{nam2015learning} trained a multi-domain CNN for learning generic representations for tracking objects. When tracking a new target, a new network is created by combining the shared layers in the pre-trained CNN with a new binary classification layer, which is online updated.
% However, even for the CNN-based trackers, the tracked box might still drift away from its true location in long-term tracking because they mostly utilize the object appearance information from the video without semantic understanding on its class.
Tracking is apparently different from VID, since it assumes the initial localization of an object in the first frame and it does not require predicting class labels.


% \vspace{0.1cm}\noindent \textbf{Object detection from video.}

% section related_work (end)

\begin{figure*}[t]
    \centering
    \includegraphics[width=0.85\linewidth]{framework}
    % \fbox{\rule{0pt}{2in} \rule{.9\linewidth}{0pt}}
    \caption{Our proposed T-CNN framework. The framework mainly consists of four components. 1) The still-image object detection component generates object region proposals in all the frames in a video clip and assigns each region proposal an initial detection score. 2) The multi-context suppression incorporates context information to suppress false positives and motion-guided propagation component utilizes motion information to propagate detection results to adjacent frames to reduce false negatives. 3) The tubelet re-scoring components utilizes tracking to obtain long bounding box sequences to enforce long-term temporal consistency of their detection scores. 4) The model combination component combines different groups of proposals and different models to generate the final results.}
    \label{fig:framework}
\end{figure*}

\section{Methods} % (fold)
\label{sec:methods}
In this section, we first introduce the VID task setting (Section~\ref{sub:task_setting}) and our overall framework (Section~\ref{sub:overview}).
Then each major component will be introduced in more details.
Section~\ref{sub:still_image} describes the settings of our still-image detectors. Section~\ref{sub:mcs_mgp} introduces how to utilize multi-context information to suppress false positive detections and utilize motion information to reduce false negatives. Global tubelet re-scoring is introduced in Section~\ref{sub:tubelet_re_scoring}.

\subsection{VID task setting} % (fold)
\label{sub:task_setting}
The ImageNet object detection from video (VID) task is similar to the object detection task (DET) in still images.
It contains $30$ classes to be detected, which are a subset of $200$ classes of the DET task. All classes are fully labeled in all the frames of each video clip.
For each video clip, algorithms need to produce a set of annotations $(f_i, c_i, s_i, b_i)$ of frame index $f_i$, class label $c_i$, confidence score $s_i$ and bounding box $b_i$.
The evaluation protocol for the VID task is the same as the DET task, i.e. we use the conventional mean average precision (Mean AP) on all classes as the evaluation metric.

\subsection{Framework overview} % (fold)
\label{sub:overview}
The proposed framework is shown in Fig.~\ref{fig:framework}.
It consists of four main components: 1) still-image detection, 2) multi-context suppression and motion-guided propagation, 3) temporal tubelet re-scoring, and 4) model combination.

\textbf{Still-image object detection.}
Our still-image object detectors adopt the DeepID-Net \cite{ouyang2015deepid} and CRAFT\cite{yang2015craft} frameworks and are trained with both ImageNet detection (DET) and video (VID) training datasets in ILSVRC 2015. DeepID-Net \cite{ouyang2015deepid} is an extension of R-CNN \cite{girshick2014rich} and CRAFT is an extension of Faster R-CNN \cite{ren2015faster}. Both of the two frameworks contain the steps of object region proposal and region proposal scoring.
The major difference is that in CRAFT (also Faster R-CNN), the proposal generation and classification are combined into a single end-to-end network.
Still-image object detectors are applied to individual frames.
The results from the two still-image object detection frameworks are treated separately for the remaining components in the proposed T-CNN framework.

\textbf{Multi-context suppression.}
This process first sorts all still-image detection scores within a video in descending orders. The classes with highly ranked detection scores are treated as high-confidence classes and the rest as low-confidence ones. The detection scores of low-confidence classes are suppressed to reduce false positives.

\textbf{Motion-guided Propagation.}
In still-image object detection, some objects may be missed in certain frames while detected in adjacent frames. Motion-guided propagation uses motion information such as optical flows to locally propagate detection results to adjacent frames to reduce false negatives.

\textbf{Temporal tubelet re-scoring.}
% For global constraints on detection results, we first run tracking algorithms starting from high-confidence detections by still-image detectors, which obtains sequences of bounding boxes or, what we call, tubelets.
Starting from high-confidence detections by still-image detectors, we first run tracking algorithms to obtain sequences of bounding boxes, which we call tubelets.
Tubelets are then classified into positive and negative samples according to the statistics of their detection scores.
Positive scores are mapped to a higher range while negative ones to a lower range, thus increasing the score margins.

\textbf{Model combination.}
For each of the two groups of proposals from DeepID-Net and CRAFT, their detection results from both tubelet re-scoring and the motion-guided propagation are each min-max mapped to $[0,1]$ and combined by an NMS process with an IOU overlap $0.5$ to obtain the final results.
% subsection overview (end)

\subsection{Still-image object detectors} % (fold)
\label{sub:still_image}
Our still-image object detectors are adopted from DeepID-Net \cite{ouyang2015deepid} and CRAFT\cite{yang2015craft}.
The two detectors have different region proposal methods, pre-trained models and training strategies.

\subsubsection{DeepID-Net}
\textbf{Object region proposals.}
For DeepID-Net, the object region proposals are obtained by selective search (SS) \cite{uijlings2013selective} and Edge Boxes (EB) \cite{Zitnick:2014edgeboxes} with a cascaded selection process that eliminates easy false positive boxes using an ImageNet pre-trained AlexNet \cite{krizhevsky2012imagenet} model.
All proposal boxes are then labeled with $200$ ImageNet detection class scores by the pre-trained AlexNet.
The boxes whose maximum prediction scores of all $200$ classes are lower than a threshold are regarded as easy negative samples and are eliminated.
The process removes around $94\%$ of all proposal boxes while obtains a recall around $90\%$.

\textbf{Pre-trained models.}
ILSVRC 2015 has two tracks for each task.
1) For the provided data track, one can use data and annotations from all ILSVRC 2015 datasets including classification and localization (CLS), DET, VID and Places2.
2) For the external data track, one can use additional data and annotations.
For the provided data track, we pretrained VGG \cite{vgg2014simonyan} and GoogLeNet \cite{googlenet} with batch normalization (BN) \cite{ioffe2015batch} using the CLS $1000$-class data, while for the external data track, we used the ImageNet $3000$-class data. Pre-training is done at the object-level annotation as in \cite{ouyang2015deepid} instead of image-level annotation in R-CNN \cite{girshick2014rich}.


\textbf{Model finetuning and SVM training.}
Since the classes in VID are a subset of DET classes, the DET pretained networks and SVM can be directly applied to the VID task, with correct class index mapping.
However, due to the mismatch of the DET and VID data distributions and the unique statistics in videos, the DET-trained models may not be optimal for the VID task.
Therefore, we finetuned the networks and re-trained the $30$ SVMs with combination of DET and VID data.
Different combination configurations are investigated and a $2:1$ DET to VID data ratio achieves the best performance (see Section~\ref{sub:parameters}).

\textbf{Score average.}
Multiple CNN and SVM models are trained separately for the DeepID-Net framework, their results are averaged to generate the detection scores.
Such score averaging process is conducted in a greedy searching manner.
The best single model is first chosen. Then for each of the remaining models, its detection scores are averaged with those of the chosen model, and the model with best performance is chosen as the second chosen model. The process repeats until no significant improvement is observed.

\subsubsection{CRAFT}

CRAFT is an extension of Faster R-CNN. It contains the Region Proposal Network (RPN) stream to generated object proposals and the Fast-RCNN stream which further assigns a class (including background) score to each proposal.

\textbf{Object region proposals.}
In this framework, we use the enhanced version of Faster-RCNN by cascade RPN and cascade Fast-RCNN.
In our cascaded version of RPN, the proposals generated by the RPN are further fed into a object/background Fast-RCNN.
We find that it leads to a $93\%$ recall rate with about $100$ proposals per image.
In our cascade version of the Fast-RCNN, we further use a class-wise softmax loss as the cascaded step.
It is utilized for hard negative mining and leads to about $2\%$ improvement in mean AP.

\textbf{Pretrained models.}
Similar to the DeepID-Net setting, the pretrained models are the VGG and GoogLeNet with batch normalization. We only use the VGG in the RPN step and use both models in the later Fast-RCNN classification step.

\textbf{Score average.} The same greedy searching is conducted for model averaging as the DeepID-Net framework.

% subsection still_image (end)

\begin{figure}[tb]
    \centering
    \includegraphics[width=0.95\linewidth]{mcs}
    % \fbox{\rule{0pt}{2in} \rule{.9\linewidth}{0pt}}
    \caption{Multi-context suppression. For each video, the classes with top detection confidences are regarded as the high-confidence classes (green arrows) and others are regarded as low-confidence ones (red arrows). The detection scores of high-confidence classes are kept the same, while those of low-confidence ones are decreased to suppress false positives.}
    \label{fig:mcs}
\end{figure}

\subsection{Multi-context suppression (MCS) and motion-guided propagation (MGP)} % (fold)
\label{sub:mcs_mgp}
\textbf{Multi-context suppression (MCS).}
One limitation of directly applying still-image object detectors to videos is that they ignore the context information within a video clip.
The detection results in each frame of a video should be strongly correlated and we can use such property to suppress false positive detections.
We observed that although video snippets in the VID dataset may contain arbitrary number of classes, statistically each video usually contains only a few classes and co-existing classes have correlations.
Statistics of all detections within a video can therefore help distinguish false positives.

For example in Fig.~\ref{fig:mcs}, in some frames from a video clip, some false positive detections have very large detection scores.
Only using the context information within these frames cannot distinguish them from the positive samples.
However, considering the detection results on other frames, we can easily determine that the majority of high-confidence detections are other classes and these positive detections are outliers.

For each frame, we have about a few hundred region proposals, each of which has detection scores of $30$ classes.
For each video clip, we rank all detection scores on all boxes in a descending order.
The classes of detection scores beyond a threshold are regarded as high-confidence classes and the rest as low-confidence classes.
The detection scores of the high-confidence classes are kept the same, while those of the low-confidence classes are suppressed by subtracting a certain value.
The threshold and subtracted value are greedily searched on the validation set.
{
We observed that most videos contain only a small number of classes. We calculated the statistics of the number of classes on each video of the training set. The mean is $\mu=1.134$ and the standard deviation is $\sigma=0.356$. %According to central limit theory, if the number of videos is large enough, the number of classes converges to normal distribution $\mathcal{N}(\alpha, \sigma^2)$.
The probability of containing more than $2$ classes in a single video clip is therefore low. The MCS selects the top classes based on the ranking of box scores and punishes the unlikely classes because of the lower possibility of a large number of classes being in the same video.
}

% subsection mcs (end)

\textbf{Motion-guided propagation (MGP).}
The multi-context suppression process can significantly reduce false positive detections, but cannot recover false negatives.
% For example in Fig.~\ref{fig:mgp}, some objects may be missed in still-image detection or may have low detection scores due to bad pose or motion blur.
The false negatives are typically caused by several reasons.
1) There are no region proposals covering enough areas of the objects;
2) Due to bad pose or motion blur of an object, its detection scores are low.

These false negatives can be recovered by adding more detections from adjacent frames, because the adjacent frames are highly correlated, the detection results should also have high correlations both in spatial locations and detection scores.
For example, if an object is still or moves at a low speed, it should appear at similar locations in adjacent frames.
This inspires us to propagate boxes and their scores of each frame to its adjacent frame to augment detections and reduce false negatives.

\begin{figure}[tb]
    \centering
    \includegraphics[width=0.95\linewidth]{mgp}
    % \fbox{\rule{0pt}{2in} \rule{.9\linewidth}{0pt}}
    \caption{Motion-guided propagation. Before the propagation, some frames may contain false negatives (e.g. some airplanes are missing in (a)). Motion-guided propagation is to propagate detections to adjacent frames (e.g. from $t=T$ to $t=T-1$ and $t=T+1$) according to the mean optical flow vector of each detection bounding box. After propagation, fewer false negatives exist in (b).}
    \label{fig:mgp}
\end{figure}

% A simple non-motion propagation can easily verify this idea by duplicating detection locations and scores to adjacent frames.
% TABLE~\ref{tab:mgp} shows the performance of different temporal durations of propagation, from which we can see that as the duration increases, the performance first increases and then decreases.
% The increase comes from augmenting more detection boxes and reduces false negatives.
% The decrease, however, are caused by location deviations from several frames away due to motion.

% Simply duplicating detections is heuristic and non-effective.

{
If the baseline still-image detector is applied to the videos frame by frame, the false negatives may be cased by missing box proposals or low-detection scores. If we assume the probability of false negatives to be $p_n$, MGP of window $w$ propagates the results of $w$ neighboring frames to the current frame, which on average reduces the mis-detection rate to $p_n^w$ if the optical flow estimation is perfect. Although, long-range MGP is limited by the accuracy of optical flow estimation, the mis-detection rate can be generally reduced within small temporal windows, and the new false positives are unlikely to be added because of the final NMS operation.
}

We propose a motion-guided approach to propagate detection bounding boxes according to the motion information.
For each region proposal, we calculate the mean optical flow vector within the bounding box of the region proposal and propagate the box coordinates with same detection scores to adjacent frames according the mean flow vectors. An illustration example is shown in Fig.~\ref{fig:mgp}.


% subsection mgp (end)

\subsection{Tubelet re-scoring} % (fold)
\label{sub:tubelet_re_scoring}
MGP generates short dense tubelets at every detection by our still-image detectors.
It significantly reduces false negatives but only incorporates short-term temporal constraints and consistency to the final detection results.
To enforce long-term temporal consistency of the results, we also need tubelets that span long periods of time.
Therefore, we use tracking algorithms to generate long tubelets and associate still-image object detections around tubelets.

As shown in Fig.~\ref{fig:framework}, the tubelet re-scoring includes three sub-steps: 1) high confidence tracking, 2) spatial max-pooling, and 3) tubelet classification.

\begin{figure}[tb]
    \centering
    \includegraphics[width=\linewidth]{temporal_rescoring}
    % \fbox{\rule{0pt}{2in} \rule{0.9\linewidth}{0pt}}
    \caption{Tubelet classification. Tubelets obtained from tracking can be classified into positive and negative samples using statistics (e.g. top-k, mean, median) of the detection scores on the tubelets. Based on the statistics on the training set, a $1$-D Bayesian classifier is trained to classify the tubelets for re-scoring.}
    \label{fig:tubelet_rescoring}
\end{figure}

\textbf{High-confidence tracking.}
For each object class in a video clip, we track high-confidence detection proposals bidirectionally over the temporal dimension.
The tracker we choose is from \cite{wang2015visual}, which in our experiments shows robust performance to different object poses and scale changes.
The starting bounding boxes of tracking are called ``anchors'', which are determined as the most confident detections.
Starting from an anchor, we track biredictionally to obtain a complete tubelet.
As the tracking is conducted along the temporal dimension, the tracked box may drift to background or other objects, or may not adapt to the scale and pose changes of the target object. Therefore, we stop the tracking early when the tracking confidence is below a threshold (probability of $0.1$ in our experiments) to reduce false positive tubelets.
After obtaining a tubelet, a new anchor is selected from the remaining detections to start a new track.
Usually, high-confidence detections tend to cluster both spatially and temporally, and therefore directly tracking the next most confident detection tends to result in tubelets with large mutual overlaps on the same object.
To reduce the redundancy and cover as many objects as possible, we perform a suppression process similar to NMS.
Detections that have overlaps with the existing tracks beyond a certain threshold (IOU, i.e. Intersection of Union, $0.3$ in our experiment) will not be chosen as new anchors.
The tracking-suppression process performs iteratively until confidence values of all remaining detections are lower than a threshold.
For each video clip, such tracking process is performed for each of the $30$ VID classes.

\textbf{Spatial max-pooling.}
After tracking, for each class, we have tubelets with high-confidence anchors. A naive approach is to classify each bounding box on the tubelets using still-image object detectors.
% In our experiment, this baseline approach has only modest performance.
Since the boxes from tracked tubelets and those from still-image object detectors have different statistics, when a still-image object detector is applied to a bounding box obtained from tracking, the detection score many not be accurate.
In addition, the tracked box locations may not be optimal due to the tracking failures. Therefore, the still-image detection scores on the tracked tubelets may not be reliable.

% To handle these problems in tubelet classification, we need to not only utilize the still-image object detection results, but also increase spatial robustness.
However, the detections spatially close to the tubelets can provide helpful information.
The spatial max-pooling process is to replace tubelet box proposals with detections of higher confidence by the still-image object detector.

For each tubelet box, we first obtain the detections from still-image object detectors that have overlaps with the box beyond a threshold (IOU $0.5$ in our setting).
Then only the detection with the maximum detection score is kept and used to replace the tracked bounding box.
This process is to simulate the conventional NMS process in object detection. If the tubelet box is indeed a positive box but with low detection score, this process can raise its detection score.
The higher the overlap threshold, the more confidence on the tubelet box.
In an extreme case when IOU = 1 is chosen as the threshold, we fully rely on the tubelet boxes while their surrounding boxes from still-image object detectors are not considered.
% We find this method effective in our experiment and different overlap threshold are evaluated in Section~\ref{sec:experiments}.

\textbf{Tubelet classification and rescoring.}
High-confidence tracking and spatial max-pooling generate long sparse tubelets that become candidates for temporal rescoring.
The main idea of temporal rescoring is to classify tubelets into positive and negative samples and map the detection scores into different ranges to increase the score margins.

Since the input only contains the original detection scores, the features for tubelet classification should also be simple.
We tried different statistics of tubelet detection scores such as mean, median and top-k (i.e. the kth largest detection score from a tubelet).
A Bayesian classifier is trained to classify the tubelets based on the statistics as shown in Fig.~\ref{fig:tubelet_rescoring}, and in our experiment, the top-k feature works best.

After classification, the detection scores of positive samples are min-max mapped to $[0.5,1]$, while negatives to $[0,0.5]$. Thus, the tubelet detection scores are globally changed so that the margins between positive and negative tubelets are increased.


% subsection tubelet_re_scoring (end)


% section methods (end)

\begin{figure*}[h]
    \centering
    \includegraphics[width=0.85\linewidth]{qualitative_results}
    \caption{Qualitative results. The bounding boxes are tight to objects because of the combination of different region proposals. The detection results are consistent across adjacent frames thanks to motion-guided propagation and tracking. The false positives are much eliminated by multi-context suppression. (Different colors are used to mark bounding boxes in the same frame and do not represent tracking results)}
    \label{fig:qualitative}
\end{figure*}

\section{Experiments} % (fold)
\label{sec:experiments}
\subsection{Dataset} % (fold)
\label{sub:dataset}

\textbf{ImageNet VID Dataset}
% We utilize the ImageNet object detection from video (VID) task dataset to evaluate the overall pipeline and individual component of the proposed framework. Up to now, only the first batch of data is released to the public.
The proposed framework is evaluated on the ImageNet VID dataset for object detection in videos.
1) The training set contains $3862$ fully-annotated video snippets ranging from $6$ frames to $5492$ frames per snippet.
2) The validation set contains $555$ fully-annotated video snippets ranging from $11$ frames to $2898$ frame per snippet.
3) The test set contains $937$ snippets and the ground truth annotation are not publicly available.
Since the official test server is primarily used for competition and has usage limitations, we primarily report the performances on the validation set as a common convention for object detection tasks. In the end, test results from top-ranked teams participated in ILSVRC 2015 are reported.

\textbf{YouTubeObjects (YTO) Dataset}
In addition to object detection on the ImageNet VID dataset, we also evaluated the object localization performance on the YTO dataset.
This dataset contains $10$ object classes with $870$ training clips and $334$ test clips.
There are several differences between ImageNet VID and YTO datasets.
1) VID dataset is fully annotated for all frames on both training and test sets, while YTO only contains bounding box annotations on very sparse frames ($4,306$ out of $521,831$ training frames and $1,781$ out of $198,321$ test frames).
2) The evaluation metric for VID is Mean AP and the system needs to detect every object in every frame. The YTO dataset, however, requires localizing objects of pre-known class for each video.

% subsection dataset (end)


\subsection{Parameter Settings} % (fold)
\label{sub:parameters}
\textbf{Data configuration.}
We investigated the ratio of training data combination from the DET and VID training sets, and its influence on the still-image object detector DeepID-Net.
The best data configuration is then used for both DeepID-Net and CRAFT.
Because the VID training set has many more fully annotated frames than the DET training set, we kept all the DET images and sampled the training frames in VID for different combination ratios in order to training the still-image object detectors.
%We evaluated different data configurations under the same SVM setting (only the positive and negative samples from DET).
%Due to the availability of the VID dataset, these early experiments are conducted on the initial release of the VID data.

We investigated several training data configurations by finetuning a GoogLeNet with BN layers. From the TABLE~\ref{tab:data_config} and \ref{tab:data_config_craft}, we can see that the ratio of $2:1$ between DET and VID data has the best performance on the still-image detector DeepID-Net and CRAFT single models, therefore, we finetuned all of our models using this data configuration.

\begin{table}[!t]
    \caption{Performances of the still-image object detector DeepID-Net single model by using different finetuning data configurations on the initial validation set. The baseline DeepID-Net of only using the DET training data has the mean AP $49.8$.}
    \label{tab:data_config}
    \centering
    \normalsize

    \begin{tabular}{c|c|c|c|c|c}
    \hline
    \hline
    \textbf{DET:VID Ratio} & 1:0 & 3:1 & 2:1 & 1:1 & 1:3 \\
    \hline
        Mean AP / \% & 49.8 & 56.9 & \textbf{58.2} & 57.6 & 57.1\\
    \hline
    \hline
    \end{tabular}
\end{table}

\begin{table}[!t]
    \caption{Performances of the still-image object detector CRAFT single model by using different finetuning data configurations on the final validation set.}
    \label{tab:data_config_craft}
    \centering
    \normalsize

    \begin{tabular}{c|c|c}
    \hline
    \hline
    \textbf{DET:VID Ratio} & 0:1 & 2:1 \\
    \hline
        Mean AP / \% & 61.5 & 63.9\\
    \hline
    \hline
    \end{tabular}
\end{table}


\begin{table}[!t]
    \caption{Performances of different data configurations on the validation set for training SVMs in DeepID-Net. Baseline (the first column) only uses DET positive and negative samples and the result is a mean AP of $49.8$.}
    \label{tab:svm_config}
    \centering

    \begin{tabular}{c|c|c|c|c|c|c}
    \hline
    \hline
    DET Positive & \cmark & \cmark & \xmark & \xmark & \xmark & \cmark \\
    VID Positive & \xmark & \cmark & \cmark & \cmark & \cmark & \cmark \\
    DET Negative & \cmark & \cmark & \cmark & \cmark & \xmark & \cmark \\
    VID Negative & \xmark & \xmark & \xmark & \cmark & \cmark & \cmark \\
    \hline
    mean AP / \%  & 49.8   & 47.1   & 35.8   & 51.6   & 52.3   & \textbf{53.7}   \\
    \hline
    \hline
    \end{tabular}
\end{table}

\begin{table}[!h]
    \caption{Performances on the validation set by different temporal window sizes of MGP.}
    \label{tab:mgp}
    \centering

    \begin{tabular}{c|c|c|c|c|c}
    \hline
    \hline
    \multirow{2}{*}{Methods} & \multicolumn{5}{c}{Temporal window size} \\
    \cline{2-6}
    & 1 (baseline) & 3 & 5 & 7 & 9 \\
    \hline
    Duplicate & \multirow{2}{*}{70.7} & 71.7 & 72.1 & 71.5 & - \\
    \cline{1-1}
    \cline{3-6}
    Motion-guided &  & 71.7 & \textbf{73.0} & \textbf{73.0} & 72.6 \\
    \hline
    \hline
    \end{tabular}
\end{table}

\begin{table*}[!t]
    \caption{Performances of individual components, frameworks and our overall system.}
    \label{tab:component_results}
    \centering
    \small{

    \begin{tabular}{c|c|c|c|c|c|c|c|c|c}
    \hline
    \hline
    \multirow{2}{*}{Data} & \multirow{2}{*}{Model} & \multirow{2}{*}{Still-image} & \multirow{2}{*}{{+MGP}} & {+MGP} & {+MGP+MCS} & Model & \multirow{2}{*}{Test Set} & Rank in  & \multirow{2}{*}{\#win} \\
    & & & & {+MCS} & {+Rescoring} & Combination & & ILSVRC2015 & \\
    \hline
    \multirow{2}{*}{Provided} & CRAFT\cite{yang2015craft} & 67.7 & {68.7} & {72.4} & {73.6} & \multirow{2}{*}{73.8} & \multirow{2}{*}{67.8} & \multirow{2}{*}{\#1} & \multirow{2}{*}{28/30} \\
    \cline{2-6}
    & DeepID-net \cite{ouyang2015deepid} & 65.8 & {68.3} & {72.1} & {72.5} & & & &\\
    \hline
    \multirow{2}{*}{Additional} & CRAFT\cite{yang2015craft} & 69.5 & {70.3} & {74.1} & {75.0} & \multirow{2}{*}{77.0} & \multirow{2}{*}{69.7} & \multirow{2}{*}{\#2} & \multirow{2}{*}{11/30} \\
    \cline{2-6}
    & DeepID-net \cite{ouyang2015deepid} & 70.7 & {72.7} & {74.9} & {75.4} & & & &\\
    \hline
    \hline
    \end{tabular}
    }
\end{table*}

% %%%%%%%%%%%%% TABLE MEAN AP %%%%%%%%%%%%%%%%%%
\begin{table*}[!t]
\caption{Performaces of our final models on the validation set.}
\label{tab:meanap}
\centering
\scriptsize{
\begin{tabular}{p{1.3cm}|p{0.5cm}p{0.5cm}p{0.5cm}p{0.5cm}p{0.5cm}p{0.5cm}p{0.5cm}p{0.5cm}p{0.5cm}p{0.5cm}p{0.5cm}p{0.5cm}p{0.5cm}p{0.5cm}p{0.5cm}p{0.5cm}}
\hline
\hline
Data & \rotatebox{60}{airplane} &  \rotatebox{60}{antelope} &  \rotatebox{60}{bear} &  \rotatebox{60}{bicycle} &  \rotatebox{60}{bird} &  \rotatebox{60}{bus} &  \rotatebox{60}{car} &  \rotatebox{60}{cattle} &  \rotatebox{60}{dog} &  \rotatebox{60}{d\_cat} &  \rotatebox{60}{elephant} &  \rotatebox{60}{fox} &  \rotatebox{60}{g\_panda} &  \rotatebox{60}{hamster} &  \rotatebox{60}{horse} &  \rotatebox{60}{lion} \\
\hline
Provided & 83.70 &  85.70 &  84.40 &  74.50 &  73.80 &  75.70 &  57.10 &  58.70 &  72.30 &  69.20 &  80.20 &  83.40 &  80.50 &  93.10 &  84.20 &  67.80 \\
\hline
Additional & 85.90 &  86.90 &  87.80 &  77.90 &  74.70 &  77.50 &  59.00 &  70.90 &  74.40 &  79.60 &  80.40 &  83.90 &  82.40 &  95.80 &  87.80 &  64.10 \\
\hline
\hline
Data & \rotatebox{60}{lizard} &  \rotatebox{60}{monkey} &  \rotatebox{60}{motorcycle} &  \rotatebox{60}{rabbit} &  \rotatebox{60}{r\_panda} &  \rotatebox{60}{sheep} &  \rotatebox{60}{snake} &  \rotatebox{60}{squirrel} &  \rotatebox{60}{tiger} &  \rotatebox{60}{train} &  \rotatebox{60}{turtle} &  \rotatebox{60}{watercraft} &  \rotatebox{60}{whale} &  \rotatebox{60}{zebra} &  \rotatebox{60}{mean AP} & \rotatebox{60}{\#win} \\
\hline
Provided & 80.30 &  54.80 &  80.60 &  63.70 &  85.70 &  60.50 &  72.90 &  52.70 &  89.70 &  81.30 &  73.70 &  69.50 &  33.50 &  90.20 &  \textbf{73.80} & 28/30 \\
\hline
Additional & 82.90 &  57.20 &  81.60 &  77.50 &  79.70 &  68.00 &  77.70 &  58.30 &  90.10 &  85.30 &  75.90 &  71.20 &  43.20 &  91.70 &  \textbf{77.00} & 11/30 \\
\hline
\hline
\end{tabular}
}
\end{table*}

In addition to model finetuning, we also investigated the data configurations for training the SVMs in DeepID-Net.
The performances are shown in TABLE~\ref{tab:svm_config}, which show that using positive and negative samples from both DET and VID data leads to the best performance.


% Using all positive and negative samples from both DET and VID datasets obtains the best performances and this setting is therefore chosen for SVM training.



Because of the redundancy among video frames, we also sampled the video frames by a factor of $2$ during testing and applied the still-image detectors to the remaining frames. The MCS, MGP and re-scoring steps in Section~\ref{sub:mcs_mgp} and \ref{sub:tubelet_re_scoring} are then conducted.
The detection boxes on the unsampled frames are generated by interpolation and MGP.
We did not observe significant performance differences with frame sampling on the validation set.

To conclude, we sampled VID frames to half the amount of DET images and combined the samples to finetune the CNN models in both DeepID-Net and CRAFT. Positive and negative samples from both DET and VID images are used to train SVMs in DeepID-Net.

\textbf{Hyperparameter settings.}
For motion-guided propagations, as described in Section~\ref{sub:mcs_mgp}, TABLE~\ref{tab:mgp} shows the performances of different propagation window sizes.
Compared to directly duplicating boxes to adjacent frames without changing their locations according to optical flow vectors, MGP has better performances with the same propagation windows, which proves that MGP generates detections with more accurate locations. $7$ frames ($3$ frames forward and $3$ backward) are empirically set as the window size.



In multi-context suppression, classes in the top $0.0003$ of all the bounding boxes in a video are regarded as high-confidence classes and the detection scores for both frameworks are subtracted by $0.4$.
Those hyperparameters are greedily searched in the validation set.


\textbf{Network configurations.}
The models in DeepID-Net and CRAFT are mainly based on GoogLeNet with batch-normalization layers and VGG models.
The techniques of multi-scale \cite{zeng2015window} and multi-region \cite{gidaris2015object} are used to further increase the number of models for score averaging in the still-image object detection shown in Fig.~\ref{fig:framework}.
The performance of a baseline DeepID-Net trained on ImageNet DET task can be increased from $49.8$ to $70.7$ with all the above-mentioned techniques (data configuration for finetuning, multi-scale, multi-region, score average, etc.).

%For region proposals from Selective Search and Edge Boxes used in DeepID-Net, the results are the averaged from $4$ types pf models including 1) a baseline GoogLeNet with BN, 2) a $4$-branch GoogLeNet with different context ranges as in [], 3) an $8$-branch GoogLeNet with multiple crops as in [], and 4) a VGG model.
%For proposals using CRAFT, the model is primarily based on a VGG-19 model and a GoogLeNet with BN layers.

% subsection parameters (end)

% section experiments (end)



\section{Results} % (fold)
\label{sec:results}

\subsection{Results on the ImageNet VID dataset} % (fold)
\label{sub:imagenet_vid_res}
\textbf{Qualitative results.} % (fold)
Some qualitative results of our proposed framework are shown in Fig.~\ref{fig:qualitative}.
From the figure, we can see the following characteristics of our proposed framework.
1) The bounding boxes are very tight to the objects, which results from the high-quality bonding box proposals combined from Selective Search, Edge Boxes and Region-proposal Networks.
2) The detections are consistent across adjacent frames without obvious false negatives thanks to the motion-guided propagation and tracking.
3) There are no obvious false positives even though the scenes may be complex (e.g. cases in the third row), because the multi-context information is used to suppress their scores.
% subsection quali_res (end)
% section experiments (end)



\textbf{Quantitative results} % (fold)
The component analysis of our framework on both provided-data and additional-data tracks are shown in TABLE~\ref{tab:component_results}. The results are obtained from the validation set. From the table, we can see that the still-image object detectors obtain about $65-70\%$ Mean AP. Adding temporal and contextual information through MCS, MGP and tubelet re-scoring significantly improves the results by up to $6.7$ percents. {As shown in TABLE~\ref{tab:component_results}, the MGP generally has $0.8$-$2.5\%$ improvement, MCS has $2.2$-$3.8\%$ improvement, and tubelet rescoring has about $0.4\%$-$1.2\%$ improvement for different models.} The final model combination process further improves the performance.

Overall, our framework ranks $1$st on the provided-data track in ILSVRC2015 winning $28$ classes out of $30$ and $2$nd on the additonal-data track winning $11$ classes. The detailed AP lists of the submitted models on the validation set are shown in TABLE~\ref{tab:meanap}. The final results of our team and other top-ranked teams on the test data are shown in TABLE~\ref{tab:offcial_results}.

\begin{table}[!h]
    \caption{Performance comparison with other teams on ILSVRC2015 VID test set with provided data (sorted by mean AP, the best model is chosen for each team).}
    \label{tab:offcial_results}
    \centering
    \normalsize

    \begin{tabular}{c|c|c|c}
    \hline
    \hline
    \textbf{Rank} & \textbf{Team name} & \textbf{mean AP} & \textbf{\#win} \\
    \hline
    \hline
    \textbf{1} & \textbf{CUVideo (Ours)} & \textbf{67.82} & \textbf{28} \\
    \hline
    2 & ITLab VID - Inha & 51.50 & 0 \\
    \hline
    3 & UIUC-IFP\cite{han2016seq} & 48.72 & 0 \\
    \hline
    4 & Trimps-Soushen & 46.12 & 0 \\
    \hline
    5 & 1-HKUST & 42.11 & 0 \\
    \hline
    6 & HiVision & 37.52 & 0 \\
    \hline
    7 & RUC\_BDAI & 35.97 & 2 \\
    \hline
    \hline
    \end{tabular}
\end{table}

{We also used the same framework (without tubelet re-scoring due to time limit) to participate in ILSVRC2016 VID challenge with provided training data. The results are shown in TABLE~\ref{tab:imagenet2016}. The final ensemble model for the challenge consists of 4 models, including 1) a ResNet-101 model, 2) a ResNet-269 model, 3) a ResNet-269 model with gated bi-directional (GBD) CNN \cite{zeng2016gated} multi-context structures, and 4) an Inception-ResNet-v2 model \cite{szegedy2016inception}. From the table we can see that compared to the results in ILSVRC2015, the 4-model ensemble has about $12.5\%$ ($73.8$ to $86.3$) improvement on validation set and $9.0\%$ ($67.8$ to $76.8$) improvement on the test set, thanks to the new architectures such as ResNet, GBD-net and Inception-ResNet. The MCS+MGP technique still has about $3.3\%$ improvement given the much improved baseline. Overall, the proposed framework ranked No. 2 in ILSVRC 2016 VID challenge with provided training data.}

{
\begin{table}[h]
    {\caption{4-model ensemble in ILSVRC 2016 challenge.}
    \label{tab:imagenet2016}}
    \centering
    \normalsize
{
    \begin{tabular}{c|c|c|c|c}
    \hline
    \hline
    \multirow{2}{*}{Still-image} & \multirow{2}{*}{+MCS} & \multirow{2}{*}{+MGP} & test & Rank in\\
    & & & set & ILSVRC2016\\
    \hline
    83.0 & 85.9 & 86.3 & 76.8 & \#2 \\
    \hline
    \hline
    \end{tabular}}
\end{table}
}

% subsection imagenet_vid_res (end)

{
\subsection{Results analysis} % (fold)
\label{sub:results_analysis}

\noindent\textbf{MGP in moving-camera sequences.} We evaluated the MGP technique in moving camera sequences. The validation videos are manually labeled into two subsets based on whether the video contains significant camera motion. In fact, the ImageNet VID dataset mainly consists of online videos, most of which are shot with hand-held cameras. The \textit{moving-camera} subset contains $401$ out of $555$ total validation videos. The mean APs of the DeepID-net model are shown in TABLE~\ref{tab:moving_camera}, from which we can see the improvement of MGP is consistent on the moving-camera subset.
One interesting observation is that the MGP has better improvement on the moving-camera subset than the full set. The reason is that usually for moving-camera sequences, the camera moves to focus on the objects of interest. Therefore, although the background is moving, the objects are relatively stable in the scenes.
% In still-camera sequences, however, the objects are usually moving, which are counter-intuitively more challenging.

\begin{table}[h]
    {\caption{MGP of DeepID-net on moving-camera subset.}
    \label{tab:moving_camera}}
    \centering
    \normalsize
{
    \begin{tabular}{c|c|c}
    \hline
    \hline
    \textbf{Set} & \textbf{still-image} & \textbf{+MGP} \\
    \hline
    \textbf{Full validation} & 70.7 & 72.7 \\
    \hline
    \textbf{Moving-camera subset} & 72.3 & 74.5 \\
    \hline
    \hline
    \end{tabular}}
\end{table}

\noindent\textbf{Time efficiency.} The proposed techniques add an limited amount of computation time to the still-image baseline. For MGP, the main additional computation is the optical flow generation. We adopted the Gunnar Farneback's algorithm in OpenCV and the speed is about $10$ fps. For MCS, the parameters are greedily searched using the detection scores, which requires little additional computation. For tubelet re-scoring, the bottleneck is the single-object tracking. The tracker that we use is a deep neural network tracker of $0.5$ fps. Since we only generates about $10$ tubelets for each video, it takes about $3$ days for completing the tracking for both training and validation set, which consists of $4417$ videos. In total, with a $8$-GPU server, it takes about $1$ week to pre-train a model on ImageNet classification task, $2$ days for fine-tuning on DET dataset, and $2$ days for fine-tuning on the VID dataset. The test time takes about $4$ days in total ($1$ day for generating still-image detections, $3$ days for incorporating temporal and context information). Considering there are a total of $3862$ training video, $555$ validation videos and $937$ test videos in the dataset with over $1$ million frames, the computational cost is acceptable.
}% color red


% subsection results_analysis (end)

\subsection{Localication on YouTubeObjects (YTO) Dataset} % (fold)
\label{sub:yto_res}
In addition to the detection results on the ImageNet VID dataset, we also evaluated our framework on the YouTubeObjects dataset for the object localization task.
Different from the object detection task, object localization requires the system to localize the objects of certain categories on certain video frames.
Since the classes of the YTO dataset is a subset of ImageNet VID dataset, the system is directly evaluated on the YTO dataset without any fine-tuning.
For each test frame, we only keep the detection result with the maximum detection score.
The evaluation metric is CorLoc \cite{Deselaers:2010localizing} as a common practice on YTO.
The CorLoc metric is the percentage of images in which the system correctly localizes one object of the target class with an overlap of IOU $>0.5$.

\begin{table*}[h]
    \caption{Object localization performances on the YTO dataset}
    \label{tab:yto_scores}
    \centering
    % \tiny
    \begin{tabular}{l|p{0.5cm}p{0.5cm}p{0.5cm}p{0.5cm}p{0.5cm}p{0.5cm}p{0.5cm}p{0.5cm}p{0.5cm}p{0.5cm}|c}
    \hline
    \hline
    Method & aero & bird & boat & car & cat & cow & dog & horse & mbike & train & Avg. \\
    \hline
    \hline
    Prest \textit{et al.} \cite{Prest:2012learning} & 51.7 & 17.5 & 34.4 & 34.7 & 22.3 & 17.9 & 13.5 & 26.7 & 41.2 & 25.0 & 28.5 \\
    Joulin \textit{et al.} \cite{Joulin:2014efficient} & 25.1 & 31.2 & 27.8 & 38.5 & 41.2 & 28.4 & 33.9 & 35.6 & 23.1 & 25.0 & 31.0 \\
    Kwak \textit{et al.} \cite{Kwak:2015unsupervised} & 56.5 & 66.4 & 58.0 & 76.8 & 39.9 & 69.3 & 50.4 & 56.3 & 53.0 & 31.0 & 55.7 \\
    Kang \textit{et al.} \cite{Kang:2016object} & 94.1 & 69.7 & 88.2 & 79.3 & 76.6 & 18.6 & 89.6 & 89.0 & 87.3 & 75.3 & 76.8 \\
    \hline
    Baseline & 91.2 & 98.1 & 85.4 & 95.0 & 92.2 & 100.0 & 96.3 & 92.8 & 91.1 & 83.0 & 92.5\\
    Baseline + MGP (window 3) & 91.8 & 98.7 & 85.4 & 95.0 & 92.2 & 100.0 & 95.7 & 93.4 & 93.9 & 84.2 & \textbf{93.0}\\
    Baseline + MGP (window 5) & 91.8 & 98.1 & 86.1 & 94.2 & 90.1 & 99.3 & 93.9 & 93.4 & 92.7 & 86.1 & 92.6\\
    \hline
    \hline
    \end{tabular}
\end{table*}


The localization performances of our system and some of the state-of-the-art results on the YTO dataset are shown in TABLE~\ref{tab:yto_scores}.
The ``baseline'' network we used for this task is the best CRAFT single model finetuned with DET and VID data of ratio $2:1$. Because only the detection with the highest score in each frame is kept, we could only apply MGP post-processing to the baseline network to improve its performance (denoted as ``baseline + MGP'').
From the table we can see that our proposed framework outperforms previous methods with large margin. Even compared to the state-of-the-art method \cite{Kang:2016object}, our system still obtains about $15$ percents improvement.
We also compared with MGP post-processing with window size of $3$ and $5$. From the table,  we can see that the window size $3$ improves the performance about $0.5$ percent, while increasing the window size to $5$ causes performance drop.
%The window $3$ has the best performance.



% subsection yto_res (end)

\section{Conclusion} % (fold)
\label{sec:conclusion}
In this work, we propose a deep learning framework that incorporates temporal and contextual information into object detection in videos.
This framework achieved the state-of-the-art performance on the ImageNet object detection from video task and won the corresponding VID challenge with provided data in ILSVRC2015.
The component analysis is investigated and discussed in details.
Code is publicly available.

The VID task is still new and under-explored.
Our proposed framework is based on the popular still-image object detection frameworks and adds important components specifically designed for videos.
We believe that the knowledge of these components can be further incorporated in to end-to-end systems and is our future research direction.


% section conclusion (end)


% if have a single appendix:
%\appendix[Proof of the Zonklar Equations]
% or
%\appendix  % for no appendix heading
% do not use \section anymore after \appendix, only \section*
% is possibly needed

% use appendices with more than one appendix
% then use \section to start each appendix
% you must declare a \section before using any
% \subsection or using \label (\appendices by itself
% starts a section numbered zero.)
%


% \appendices
% \section{Proof of the First Zonklar Equation}
% Appendix one text goes here.

% % you can choose not to have a title for an appendix
% % if you want by leaving the argument blank
% \section{}
% Appendix two text goes here.


% use section* for acknowledgment
% \ifCLASSOPTIONcompsoc
%   % The Computer Society usually uses the plural form
%   \section*{Acknowledgments}
% \else
%   % regular IEEE prefers the singular form
%   \section*{Acknowledgment}
% \fi


% The authors would like to thank...


% Can use something like this to put references on a page
% by themselves when using endfloat and the captionsoff option.
\ifCLASSOPTIONcaptionsoff
  \newpage
\fi



% trigger a \newpage just before the given reference
% number - used to balance the columns on the last page
% adjust value as needed - may need to be readjusted if
% the document is modified later
%\IEEEtriggeratref{8}
% The "triggered" command can be changed if desired:
%\IEEEtriggercmd{\enlargethispage{-5in}}

% references section

% can use a bibliography generated by BibTeX as a .bbl file
% BibTeX documentation can be easily obtained at:
% http://mirror.ctan.org/biblio/bibtex/contrib/doc/
% The IEEEtran BibTeX style support page is at:
% http://www.michaelshell.org/tex/ieeetran/bibtex/
% \bibliographystyle{IEEEtran}
% % argument is your BibTeX string definitions and bibliography database(s)
% \bibliography{IEEEabrv,egbib}
%
% <OR> manually copy in the resultant .bbl file
% set second argument of \begin to the number of references
% (used to reserve space for the reference number labels box)
% \begin{thebibliography}{1}

% \bibitem{IEEEhowto:kopka}
% H.~Kopka and P.~W. Daly, \emph{A Guide to \LaTeX}, 3rd~ed.\hskip 1em plus
%   0.5em minus 0.4em\relax Harlow, England: Addison-Wesley, 1999.

% \end{thebibliography}

% Generated by IEEEtran.bst, version: 1.13 (2008/09/30)
\begin{thebibliography}{10}
\providecommand{\url}[1]{#1}
\csname url@samestyle\endcsname
\providecommand{\newblock}{\relax}
\providecommand{\bibinfo}[2]{#2}
\providecommand{\BIBentrySTDinterwordspacing}{\spaceskip=0pt\relax}
\providecommand{\BIBentryALTinterwordstretchfactor}{4}
\providecommand{\BIBentryALTinterwordspacing}{\spaceskip=\fontdimen2\font plus
\BIBentryALTinterwordstretchfactor\fontdimen3\font minus
  \fontdimen4\font\relax}
\providecommand{\BIBforeignlanguage}[2]{{%
\expandafter\ifx\csname l@#1\endcsname\relax
\typeout{** WARNING: IEEEtran.bst: No hyphenation pattern has been}%
\typeout{** loaded for the language `#1'. Using the pattern for}%
\typeout{** the default language instead.}%
\else
\language=\csname l@#1\endcsname
\fi
#2}}
\providecommand{\BIBdecl}{\relax}
\BIBdecl

\bibitem{googlenet}
C.~Szegedy, W.~Liu, Y.~Jia, P.~Sermanet, S.~Reed, D.~Anguelov, D.~Erhan,
  V.~Vanhoucke, and A.~Rabinovich, ``Going deeper with convolutions,''
  \emph{CVPR}, 2015.

\bibitem{vgg2014simonyan}
K.~Simonyan and A.~Zisserman, ``Very deep convolutional networks for
  large-scale image recognition,'' \emph{Int'l Conf. Learning Representations},
  2014.

\bibitem{girshick2014rich}
R.~Girshick, J.~Donahue, T.~Darrell, and J.~Malik, ``Rich feature hierarchies
  for accurate object detection and semantic segmentation,'' \emph{CVPR}, 2014.

\bibitem{girshick2015fast}
R.~Girshick, ``Fast r-cnn,'' \emph{ICCV}, 2015.

\bibitem{ren2015faster}
S.~Ren, K.~He, R.~Girshick, and J.~Sun, ``Faster r-cnn: Towards real-time
  object detection with region proposal networks,'' \emph{NIPS}, 2015.

\bibitem{he2016deep}
K.~He, X.~Zhang, S.~Ren, and J.~Sun, ``Deep residual learning for image
  recognition,'' in \emph{Proceedings of the IEEE Conference on Computer Vision
  and Pattern Recognition}, 2016, pp. 770--778.

\bibitem{ioffe2015batch}
S.~Ioffe and C.~Szegedy, ``Batch normalization: Accelerating deep network
  training by reducing internal covariate shift,'' \emph{arXiv preprint
  arXiv:1502.03167}, 2015.

\bibitem{ouyang2015deepid}
W.~Ouyang, X.~Wang, X.~Zeng, S.~Qiu, P.~Luo, Y.~Tian, H.~Li, S.~Yang, Z.~Wang,
  C.-C. Loy \emph{et~al.}, ``Deep{ID}-net: Deformable deep convolutional neural
  networks for object detection,'' \emph{CVPR}, 2015.

\bibitem{Oneata:2014spatio}
D.~Oneata, J.~Revaud, J.~Verbeek, and C.~Schmid, ``{Spatio-temporal Object
  Detection Proposals},'' \emph{ECCV}, 2014.

\bibitem{sermanet2013overfeat}
P.~Sermanet, D.~Eigen, X.~Zhang, M.~Mathieu, R.~Fergus, and Y.~LeCun,
  ``Overfeat: Integrated recognition, localization and detection using
  convolutional networks,'' \emph{arXiv preprint arXiv:1312.6229}, 2013.

\bibitem{erhan2014scalable}
D.~Erhan, C.~Szegedy, A.~Toshev, and D.~Anguelov, ``Scalable object detection
  using deep neural networks,'' 2014.

\bibitem{redmon2016you}
J.~Redmon, S.~Divvala, R.~Girshick, and A.~Farhadi, ``{You Only Look Once:
  Unified, Real-Time Object Detection},'' in \emph{CVPR}, 2016.

\bibitem{gidaris2015object}
S.~Gidaris and N.~Komodakis, ``Object detection via a multi-region and semantic
  segmentation-aware cnn model,'' in \emph{ICCV}, 2015.

\bibitem{girshick2015deformable}
R.~Girshick, F.~Iandola, T.~Darrell, and J.~Malik, ``Deformable part models are
  convolutional neural networks,'' in \emph{CVPR}, 2015.

\bibitem{mrowca2015spatial}
D.~Mrowca, M.~Rohrbach, J.~Hoffman, R.~Hu, K.~Saenko, and T.~Darrell, ``Spatial
  semantic regularisation for large scale object detection,'' in \emph{ICCV},
  2015.

\bibitem{bell2015inside}
S.~Bell, C.~L. Zitnick, K.~Bala, and R.~Girshick, ``Inside-outside net:
  Detecting objects in context with skip pooling and recurrent neural
  networks,'' \emph{arXiv preprint arXiv:1512.04143}, 2015.

\bibitem{yan2015object}
J.~Yan, Y.~Yu, X.~Zhu, Z.~Lei, and S.~Z. Li, ``Object detection by labeling
  superpixels,'' in \emph{CVPR}, 2015.

\bibitem{wang2016dictionary}
K.~Wang, L.~Lin, W.~Zuo, S.~Gu, and L.~Zhang, ``Dictionary pair classifier
  driven convolutional neural networks for object detection,'' in \emph{CVPR},
  2016, pp. 2138--2146.

\bibitem{liu2016ssd}
W.~Liu, D.~Anguelov, D.~Erhan, C.~Szegedy, S.~Reed, C.-Y. Fu, and A.~C. Berg,
  ``Ssd: Single shot multibox detector,'' in \emph{ECCV}.\hskip 1em plus 0.5em
  minus 0.4em\relax Springer, 2016, pp. 21--37.

\bibitem{han2016seq}
W.~Han, P.~Khorrami, T.~L. Paine, P.~Ramachandran, M.~Babaeizadeh, H.~Shi,
  J.~Li, S.~Yan, and T.~S. Huang, ``Seq-nms for video object detection,''
  \emph{arXiv preprint arXiv:1602.08465}, 2016.

\bibitem{galteri2017spatio}
L.~Galteri, L.~Seidenari, M.~Bertini, and A.~Del~Bimbo, ``Spatio-temporal
  closed-loop object detection,'' \emph{IEEE Transactions on Image Processing},
  2017.

\bibitem{kang2017object}
K.~Kang, H.~Li, T.~Xiao, W.~Ouyang, J.~Yan, X.~Liu, and X.~Wang, ``Object
  detection in videos with tubelet proposal networks,'' \emph{CVPR}, 2017.

\bibitem{Prest:2012learning}
A.~Prest, C.~Leistner, J.~Civera, C.~Schmid, and V.~Ferrari, ``{Learning object
  class detectors from weakly annotated video},'' \emph{CVPR}, 2012.

\bibitem{Papazoglou:2013fast}
A.~Papazoglou and V.~Ferrari, ``{Fast Object Segmentation in Unconstrained
  Video},'' \emph{ICCV}, 2013.

\bibitem{Joulin:2014efficient}
A.~Joulin, K.~Tang, and L.~Fei-Fei, ``{Efficient Image and Video
  Co-localization with Frank-Wolfe Algorithm},'' \emph{ECCV}, 2014.

\bibitem{Kwak:2015unsupervised}
S.~Kwak, M.~Cho, I.~Laptev, J.~Ponce, and C.~Schmid, ``{Unsupervised Object
  Discovery and Tracking in Video Collections},'' \emph{ICCV}, 2015.

\bibitem{Maxime:2015is}
O.~Maxime, B.~Leon, I.~Laptev, and J.~Sivic, ``{Is object localization for
  free? {\textendash} Weakly-supervised learning with convolutional neural
  networks },'' in \emph{Computer Vision and Pattern Recognition}, 2015.

\bibitem{rubinstein2013unsupervised}
M.~Rubinstein, A.~Joulin, J.~Kopf, and C.~Liu, ``Unsupervised joint object
  discovery and segmentation in internet images,'' in \emph{CVPR}, 2013.

\bibitem{deng2009imagenet}
J.~Deng, W.~Dong, R.~Socher, L.-J. Li, K.~Li, and L.~Fei-Fei, ``Imagenet: A
  large-scale hierarchical image database,'' \emph{CVPR}, 2009.

\bibitem{krizhevsky2012imagenet}
A.~Krizhevsky, I.~Sutskever, and G.~E. Hinton, ``Imagenet classification with
  deep convolutional neural networks,'' \emph{NIPS}, 2012.

\bibitem{wang2016cost}
K.~Wang, D.~Zhang, Y.~Li, R.~Zhang, and L.~Lin, ``Cost-effective active
  learning for deep image classification,'' \emph{TCSVT}, 2016.

\bibitem{Simonyan:2014twostream}
K.~Simonyan and A.~Zisserman, ``{Two-Stream Convolutional Networks for Action
  Recognition in Videos},'' in \emph{Conference on Neural Information
  Processing Systems}, 2014.

\bibitem{long2015fully}
J.~Long, E.~Shelhamer, and T.~Darrell, ``Fully convolutional networks for
  semantic segmentation,'' in \emph{Proceedings of the IEEE Conference on
  Computer Vision and Pattern Recognition}, 2015, pp. 3431--3440.

\bibitem{possegger2014occlusion}
H.~Possegger, T.~Mauthner, P.~M. Roth, and H.~Bischof, ``Occlusion geodesics
  for online multi-object tracking,'' \emph{CVPR}, 2014.

\bibitem{li2015reliable}
Y.~Li, J.~Zhu, and S.~C. Hoi, ``Reliable patch trackers: Robust visual tracking
  by exploiting reliable patches,'' \emph{CVPR}, 2015.

\bibitem{hong2015multi}
Z.~Hong, Z.~Chen, C.~Wang, X.~Mei, D.~Prokhorov, and D.~Tao, ``Multi-store
  tracker (muster): a cognitive psychology inspired approach to object
  tracking,'' \emph{CVPR}, 2015.

\bibitem{maggio2007adaptive}
E.~Maggio, F.~Smerladi, and A.~Cavallaro, ``Adaptive multifeature tracking in a
  particle filtering framework,'' \emph{TCSVT}, 2007.

\bibitem{maggio2008efficient}
E.~Maggio, M.~Taj, and A.~Cavallaro, ``Efficient multitarget visual tracking
  using random finite sets,'' \emph{TCSVT}, 2008.

\bibitem{cavallaro2005tracking}
A.~Cavallaro, O.~Steiger, and T.~Ebrahimi, ``Tracking video objects in
  cluttered background,'' \emph{TCSVT}, 2005.

\bibitem{yang2009robust}
J.~Yang, D.~Schonfeld, and M.~Mohamed, ``Robust video stabilization based on
  particle filter tracking of projected camera motion,'' \emph{TCSVT}, 2009.

\bibitem{li2017learning}
C.~Li, L.~Lin, W.~Zuo, and J.~Tang, ``Learning patch-based dynamic graph for
  visual tracking.'' in \emph{AAAI}, 2017, pp. 4126--4132.

\bibitem{wang2015visual}
L.~Wang, W.~Ouyang, X.~Wang, and H.~Lu, ``Visual tracking with fully
  convolutional networks,'' \emph{ICCV}, 2015.

\bibitem{nam2015learning}
H.~Nam and B.~Han, ``Learning multi-domain convolutional neural networks for
  visual tracking,'' \emph{arXiv:1510.07945}, 2015.

\bibitem{yang2015craft}
B.~Yang, J.~Yan, Z.~Lei, and S.~Z. Li, ``Craft objects from images,''
  \emph{CVPR}, 2016.

\bibitem{uijlings2013selective}
J.~R. Uijlings, K.~E. van~de Sande, T.~Gevers, and A.~W. Smeulders, ``Selective
  search for object recognition,'' \emph{IJCV}, 2013.

\bibitem{Zitnick:2014edgeboxes}
C.~L. Zitnick and P.~Dollar, ``{Edge Boxes: Locating Object Proposals from
  Edges},'' \emph{ECCV}, 2014.

\bibitem{zeng2015window}
X.~Zeng, W.~Ouyang, and X.~Wang, ``Window-object relationship guided
  representation learning for generic object detections,'' \emph{arXiv preprint
  arXiv:1512.02736}, 2015.

\bibitem{zeng2016gated}
X.~Zeng, W.~Ouyang, B.~Yang, J.~Yan, and X.~Wang, ``Gated bi-directional cnn
  for object detection,'' in \emph{ECCV}.\hskip 1em plus 0.5em minus
  0.4em\relax Springer, 2016, pp. 354--369.

\bibitem{szegedy2016inception}
C.~Szegedy, S.~Ioffe, V.~Vanhoucke, and A.~Alemi, ``Inception-v4,
  inception-resnet and the impact of residual connections on learning,''
  \emph{arXiv preprint arXiv:1602.07261}, 2016.

\bibitem{Deselaers:2010localizing}
T.~Deselaers, B.~Alexe, and V.~Ferrari, ``{Localizing Objects While Learning
  Their Appearance},'' \emph{ECCV}, 2010.

\bibitem{Kang:2016object}
K.~Kang, W.~Ouyang, H.~Li, and X.~Wang, ``{Object Detection from Video Tubelets
  with Convolutional Neural Networks},'' in \emph{Computer Vision and Pattern
  Recognition}, 2016.

\end{thebibliography}


% biography section
%
% If you have an EPS/PDF photo (graphicx package needed) extra braces are
% needed around the contents of the optional argument to biography to prevent
% the LaTeX parser from getting confused when it sees the complicated
% \includegraphics command within an optional argument. (You could create
% your own custom macro containing the \includegraphics command to make things
% simpler here.)
\vspace{-1.5cm}
\begin{IEEEbiography}[{\includegraphics[width=1in,height=1.25in,clip,keepaspectratio]{kai_kang}}]{Kai Kang} graduated from School of the Gifted Young at University of Science and Technology of China (USTC) in 2013 with a BS (Hons) degree in Optics. He is currently a PhD candidate in Electronic Engineering at the Chinese University of Hong Kong (CUHK). His research interests include computer vision, video analysis and deep learning.
\end{IEEEbiography}
\vspace{-1.5cm}
\begin{IEEEbiography}[{\includegraphics[width=1in,height=1.25in,clip,keepaspectratio]{hongsheng_li}}]{Hongsheng Li} received the bachelors degree in automation from East China University of Science and Technology, and the masters and doctorate degrees in computer science from Lehigh University, Pennsylvania, in 2006, 2010, and 2012, respectively. He is a research assistant professor in the Department of Electronic Engineering at the Chinese University of Hong Kong. His research interests include computer vision, medical image analysis and machine learning.
\end{IEEEbiography}
\vspace{-1.5cm}
\begin{IEEEbiography}[{\includegraphics[width=1in,height=1.25in,clip,keepaspectratio]{junjie_yan}}]{Junjie Yan} is the R\&D Director at SenseTime Group Limited. In SenseTime, he leads the R\&D in detection, tracking, recognition, and video intelligence. He graduated in Jul. 2015 with a PhD degree at National Laboratory of Pattern Recognition, Chinese Academy of Sciences, where he mainly worked on object detection and face recognition.
\end{IEEEbiography}
\vspace{-1.5cm}
\begin{IEEEbiography}[{\includegraphics[width=1in,height=1.25in,clip,keepaspectratio]{xingyu_zeng}}]{Xingyu Zeng} received the PhD degree in the Department of Electronic Engineering from Chinese University of Hong Kong in 2016 and the BS degree in electronic engineering and information science from the University of Science and Technology, China, in 2011. He is currently working in Sensetime Group Limited. His research interests include computer vision and deep learning. 
\end{IEEEbiography}
\vspace{-1.5cm}
\begin{IEEEbiography}[{\includegraphics[width=1in,height=1.25in,clip,keepaspectratio]{bin_yang}}]{Bin Yang} received the B.Eng. degree in Electronics and Information Engineering from China Agricultural University in 2014. He is currently pursuing the M.Sc. degree with the Computer Science Department, University of Toronto. His research interests include computer vision and machine learning. 
\end{IEEEbiography}
\vspace{-1.5cm}
\begin{IEEEbiography}[{\includegraphics[width=1in,height=1.25in,clip,keepaspectratio]{tong_xiao}}]{Tong Xiao} is a Ph.D. candidate in electronic engineering at The Chinese University of Hong Kong, advised by Prof. Xiaogang Wang. Tong received Bachelor of Engineering in computer science from Tsinghua University. His research interests include computer vision and deep learning, especially learning better feature representations for human identification. He also served as a reviewer of top-tier computer vision conferences and journals, including CVPR, ICCV, ECCV, TIP, TNNLS, and TCSVT.
\end{IEEEbiography}
\vspace{-1.5cm}
\begin{IEEEbiography}[{\includegraphics[width=1in,height=1.25in,clip,keepaspectratio]{cong_zhang}}]{Cong Zhang} received the B.S. degree from Shanghai Jiao Tong University, Shanghai in 2009 from Department of Electronic Engineering. He is currently pursuing the Ph.D. degree at the Institute of Image Communication and Information Processing, Shanghai Jiao Tong University. His current research interests include crowd understanding, crowd behavior detection, and machine learning.
\end{IEEEbiography}
\vspace{-1.5cm}
\begin{IEEEbiography}[{\includegraphics[width=1in,height=1.25in,clip,keepaspectratio]{zhe_wang}}]{Zhe Wang} received the BEng from the Department of Optical Engineering of Zhejiang University, China, in 2012. He is currently working toward a PhD degree in the Department of Electronic Engineering, the Chinese University of Hong Kong. His research interest is focused on deep learning and its applications, especially on medical imaging, segmentation and detection.
\end{IEEEbiography}
\vspace{-1.5cm}
\begin{IEEEbiography}[{\includegraphics[width=1in,height=1.25in,clip,keepaspectratio]{ruohui_wang}}]{Ruohui Wang} received his B.Eng. degree from Department of Electronic Engineering at Tsinghua University in 2013. He is currently a Ph.D. student in Department of Information Engineering at the Chinese University of Hong Kong. His research interests include machine learning and distributed computing.
\end{IEEEbiography}
\vspace{-1.5cm}
\begin{IEEEbiography}[{\includegraphics[width=1in,height=1.25in,clip,keepaspectratio]{xiaogang_wang}}]{Xiaogang Wang} received the BS degree from the Special Class for Gifted Young at University of Science and Technology of China in electrical engineering and information science in 2001, and the MPhil degree from Chinese University of Hong Kong in 2004. He received the PhD degree in computer science from the Massachusetts Institute of Technology. He is currently an associate professor in the Department of Electronic Engineering at the Chinese University of Hong Kong. His research interests include computer vision and machine learning.
\end{IEEEbiography}
\vspace{-1.5cm}
\begin{IEEEbiography}[{\includegraphics[width=1in,height=1.25in,clip,keepaspectratio]{wanli_ouyang}}]{Wanli Ouyang} received the PhD degree in the Department of Electronic Engineering, The Chinese University of Hong Kong. Since 2017, he is a senior lecturer at the University of Sydney. His research interests include image processing, computer vision and pattern recognition.
\end{IEEEbiography}
% \enlargethispage{-5in}



% % if you will not have a photo at all:
% \begin{IEEEbiographynophoto}{John Doe}
% Biography text here.
% \end{IEEEbiographynophoto}

% % insert where needed to balance the two columns on the last page with
% % biographies
% %\newpage

% \begin{IEEEbiographynophoto}{Jane Doe}
% Biography text here.
% \end{IEEEbiographynophoto}

% You can push biographies down or up by placing
% a \vfill before or after them. The appropriate
% use of \vfill depends on what kind of text is
% on the last page and whether or not the columns
% are being equalized.

% \vfill

% Can be used to pull up biographies so that the bottom of the last one
% is flush with the other column.
% \enlargethispage{-5in}



% that's all folks
\end{document}



1604.02532.extracted_bib
{IEEEhowto:kopka % H.~Kopka and P.~W. Daly, A Guide to \LaTeX, 3rd~ed.\hskip 1em plus % 0.5em minus 0.4em\relax Harlow, England: Addison-Wesley, 1999. %
1604.02532.extracted_bib
{googlenet C.~Szegedy, W.~Liu, Y.~Jia, P.~Sermanet, S.~Reed, D.~Anguelov, D.~Erhan, V.~Vanhoucke, and A.~Rabinovich, ``Going deeper with convolutions,'' CVPR, 2015.
1604.02532.extracted_bib
{vgg2014simonyan K.~Simonyan and A.~Zisserman, ``Very deep convolutional networks for large-scale image recognition,'' Int'l Conf. Learning Representations, 2014.
1604.02532.extracted_bib
{girshick2014rich R.~Girshick, J.~Donahue, T.~Darrell, and J.~Malik, ``Rich feature hierarchies for accurate object detection and semantic segmentation,'' CVPR, 2014.
1604.02532.extracted_bib
{girshick2015fast R.~Girshick, ``Fast r-cnn,'' ICCV, 2015.
1604.02532.extracted_bib
{ren2015faster S.~Ren, K.~He, R.~Girshick, and J.~Sun, ``Faster r-cnn: Towards real-time object detection with region proposal networks,'' NIPS, 2015.
1604.02532.extracted_bib
{he2016deep K.~He, X.~Zhang, S.~Ren, and J.~Sun, ``Deep residual learning for image recognition,'' in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016, pp. 770--778.
1604.02532.extracted_bib
{ioffe2015batch S.~Ioffe and C.~Szegedy, ``Batch normalization: Accelerating deep network training by reducing internal covariate shift,'' arXiv preprint arXiv:1502.03167, 2015.
1604.02532.extracted_bib
{ouyang2015deepid W.~Ouyang, X.~Wang, X.~Zeng, S.~Qiu, P.~Luo, Y.~Tian, H.~Li, S.~Yang, Z.~Wang, C.-C. Loy et~al., ``Deep{ID-net: Deformable deep convolutional neural networks for object detection,'' CVPR, 2015.
1604.02532.extracted_bib
{Oneata:2014spatio D.~Oneata, J.~Revaud, J.~Verbeek, and C.~Schmid, ``{Spatio-temporal Object Detection Proposals,'' ECCV, 2014.
1604.02532.extracted_bib
{sermanet2013overfeat P.~Sermanet, D.~Eigen, X.~Zhang, M.~Mathieu, R.~Fergus, and Y.~LeCun, ``Overfeat: Integrated recognition, localization and detection using convolutional networks,'' arXiv preprint arXiv:1312.6229, 2013.
1604.02532.extracted_bib
{erhan2014scalable D.~Erhan, C.~Szegedy, A.~Toshev, and D.~Anguelov, ``Scalable object detection using deep neural networks,'' 2014.
1604.02532.extracted_bib
{redmon2016you J.~Redmon, S.~Divvala, R.~Girshick, and A.~Farhadi, ``{You Only Look Once: Unified, Real-Time Object Detection,'' in CVPR, 2016.
1604.02532.extracted_bib
{gidaris2015object S.~Gidaris and N.~Komodakis, ``Object detection via a multi-region and semantic segmentation-aware cnn model,'' in ICCV, 2015.
1604.02532.extracted_bib
{girshick2015deformable R.~Girshick, F.~Iandola, T.~Darrell, and J.~Malik, ``Deformable part models are convolutional neural networks,'' in CVPR, 2015.
1604.02532.extracted_bib
{mrowca2015spatial D.~Mrowca, M.~Rohrbach, J.~Hoffman, R.~Hu, K.~Saenko, and T.~Darrell, ``Spatial semantic regularisation for large scale object detection,'' in ICCV, 2015.
1604.02532.extracted_bib
{bell2015inside S.~Bell, C.~L. Zitnick, K.~Bala, and R.~Girshick, ``Inside-outside net: Detecting objects in context with skip pooling and recurrent neural networks,'' arXiv preprint arXiv:1512.04143, 2015.
1604.02532.extracted_bib
{yan2015object J.~Yan, Y.~Yu, X.~Zhu, Z.~Lei, and S.~Z. Li, ``Object detection by labeling superpixels,'' in CVPR, 2015.
1604.02532.extracted_bib
{wang2016dictionary K.~Wang, L.~Lin, W.~Zuo, S.~Gu, and L.~Zhang, ``Dictionary pair classifier driven convolutional neural networks for object detection,'' in CVPR, 2016, pp. 2138--2146.
1604.02532.extracted_bib
{liu2016ssd W.~Liu, D.~Anguelov, D.~Erhan, C.~Szegedy, S.~Reed, C.-Y. Fu, and A.~C. Berg, ``Ssd: Single shot multibox detector,'' in ECCV.\hskip 1em plus 0.5em minus 0.4em\relax Springer, 2016, pp. 21--37.
1604.02532.extracted_bib
{han2016seq W.~Han, P.~Khorrami, T.~L. Paine, P.~Ramachandran, M.~Babaeizadeh, H.~Shi, J.~Li, S.~Yan, and T.~S. Huang, ``Seq-nms for video object detection,'' arXiv preprint arXiv:1602.08465, 2016.
1604.02532.extracted_bib
{galteri2017spatio L.~Galteri, L.~Seidenari, M.~Bertini, and A.~Del~Bimbo, ``Spatio-temporal closed-loop object detection,'' IEEE Transactions on Image Processing, 2017.
1604.02532.extracted_bib
{kang2017object K.~Kang, H.~Li, T.~Xiao, W.~Ouyang, J.~Yan, X.~Liu, and X.~Wang, ``Object detection in videos with tubelet proposal networks,'' CVPR, 2017.
1604.02532.extracted_bib
{Prest:2012learning A.~Prest, C.~Leistner, J.~Civera, C.~Schmid, and V.~Ferrari, ``{Learning object class detectors from weakly annotated video,'' CVPR, 2012.
1604.02532.extracted_bib
{Papazoglou:2013fast A.~Papazoglou and V.~Ferrari, ``{Fast Object Segmentation in Unconstrained Video,'' ICCV, 2013.
1604.02532.extracted_bib
{Joulin:2014efficient A.~Joulin, K.~Tang, and L.~Fei-Fei, ``{Efficient Image and Video Co-localization with Frank-Wolfe Algorithm,'' ECCV, 2014.
1604.02532.extracted_bib
{Kwak:2015unsupervised S.~Kwak, M.~Cho, I.~Laptev, J.~Ponce, and C.~Schmid, ``{Unsupervised Object Discovery and Tracking in Video Collections,'' ICCV, 2015.
1604.02532.extracted_bib
{Maxime:2015is O.~Maxime, B.~Leon, I.~Laptev, and J.~Sivic, ``{Is object localization for free? {\textendash Weakly-supervised learning with convolutional neural networks ,'' in Computer Vision and Pattern Recognition, 2015.
1604.02532.extracted_bib
{rubinstein2013unsupervised M.~Rubinstein, A.~Joulin, J.~Kopf, and C.~Liu, ``Unsupervised joint object discovery and segmentation in internet images,'' in CVPR, 2013.
1604.02532.extracted_bib
{deng2009imagenet J.~Deng, W.~Dong, R.~Socher, L.-J. Li, K.~Li, and L.~Fei-Fei, ``Imagenet: A large-scale hierarchical image database,'' CVPR, 2009.
1604.02532.extracted_bib
{krizhevsky2012imagenet A.~Krizhevsky, I.~Sutskever, and G.~E. Hinton, ``Imagenet classification with deep convolutional neural networks,'' NIPS, 2012.
1604.02532.extracted_bib
{wang2016cost K.~Wang, D.~Zhang, Y.~Li, R.~Zhang, and L.~Lin, ``Cost-effective active learning for deep image classification,'' TCSVT, 2016.
1604.02532.extracted_bib
{Simonyan:2014twostream K.~Simonyan and A.~Zisserman, ``{Two-Stream Convolutional Networks for Action Recognition in Videos,'' in Conference on Neural Information Processing Systems, 2014.
1604.02532.extracted_bib
{long2015fully J.~Long, E.~Shelhamer, and T.~Darrell, ``Fully convolutional networks for semantic segmentation,'' in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2015, pp. 3431--3440.
1604.02532.extracted_bib
{possegger2014occlusion H.~Possegger, T.~Mauthner, P.~M. Roth, and H.~Bischof, ``Occlusion geodesics for online multi-object tracking,'' CVPR, 2014.
1604.02532.extracted_bib
{li2015reliable Y.~Li, J.~Zhu, and S.~C. Hoi, ``Reliable patch trackers: Robust visual tracking by exploiting reliable patches,'' CVPR, 2015.
1604.02532.extracted_bib
{hong2015multi Z.~Hong, Z.~Chen, C.~Wang, X.~Mei, D.~Prokhorov, and D.~Tao, ``Multi-store tracker (muster): a cognitive psychology inspired approach to object tracking,'' CVPR, 2015.
1604.02532.extracted_bib
{maggio2007adaptive E.~Maggio, F.~Smerladi, and A.~Cavallaro, ``Adaptive multifeature tracking in a particle filtering framework,'' TCSVT, 2007.
1604.02532.extracted_bib
{maggio2008efficient E.~Maggio, M.~Taj, and A.~Cavallaro, ``Efficient multitarget visual tracking using random finite sets,'' TCSVT, 2008.
1604.02532.extracted_bib
{cavallaro2005tracking A.~Cavallaro, O.~Steiger, and T.~Ebrahimi, ``Tracking video objects in cluttered background,'' TCSVT, 2005.
1604.02532.extracted_bib
{yang2009robust J.~Yang, D.~Schonfeld, and M.~Mohamed, ``Robust video stabilization based on particle filter tracking of projected camera motion,'' TCSVT, 2009.
1604.02532.extracted_bib
{li2017learning C.~Li, L.~Lin, W.~Zuo, and J.~Tang, ``Learning patch-based dynamic graph for visual tracking.'' in AAAI, 2017, pp. 4126--4132.
1604.02532.extracted_bib
{wang2015visual L.~Wang, W.~Ouyang, X.~Wang, and H.~Lu, ``Visual tracking with fully convolutional networks,'' ICCV, 2015.
1604.02532.extracted_bib
{nam2015learning H.~Nam and B.~Han, ``Learning multi-domain convolutional neural networks for visual tracking,'' arXiv:1510.07945, 2015.
1604.02532.extracted_bib
{yang2015craft B.~Yang, J.~Yan, Z.~Lei, and S.~Z. Li, ``Craft objects from images,'' CVPR, 2016.
1604.02532.extracted_bib
{uijlings2013selective J.~R. Uijlings, K.~E. van~de Sande, T.~Gevers, and A.~W. Smeulders, ``Selective search for object recognition,'' IJCV, 2013.
1604.02532.extracted_bib
{Zitnick:2014edgeboxes C.~L. Zitnick and P.~Dollar, ``{Edge Boxes: Locating Object Proposals from Edges,'' ECCV, 2014.
1604.02532.extracted_bib
{zeng2015window X.~Zeng, W.~Ouyang, and X.~Wang, ``Window-object relationship guided representation learning for generic object detections,'' arXiv preprint arXiv:1512.02736, 2015.
1604.02532.extracted_bib
{zeng2016gated X.~Zeng, W.~Ouyang, B.~Yang, J.~Yan, and X.~Wang, ``Gated bi-directional cnn for object detection,'' in ECCV.\hskip 1em plus 0.5em minus 0.4em\relax Springer, 2016, pp. 354--369.
1604.02532.extracted_bib
{szegedy2016inception C.~Szegedy, S.~Ioffe, V.~Vanhoucke, and A.~Alemi, ``Inception-v4, inception-resnet and the impact of residual connections on learning,'' arXiv preprint arXiv:1602.07261, 2016.
1604.02532.extracted_bib
{Deselaers:2010localizing T.~Deselaers, B.~Alexe, and V.~Ferrari, ``{Localizing Objects While Learning Their Appearance,'' ECCV, 2010.
1604.02532.extracted_bib
{Kang:2016object K.~Kang, W.~Ouyang, H.~Li, and X.~Wang, ``{Object Detection from Video Tubelets with Convolutional Neural Networks,'' in Computer Vision and Pattern Recognition, 2016.