The ImageNet Shuffle:\\Reorganized Pre-training for Video Event Detection
% You can go ahead and credit any number of authors here
% e.g. one 'row of three' or two rows (consisting of one row of three % and a second row of one
two or three). % % The command \alignauthor (no curly braces needed) should % precede each author name
affiliation/snail-mail address and % e-mail address. Additionally
tag each line of % affiliation/address with \affaddr
and tag the % e-mail address with \email. % % 1st. author %\alignauthor \numberofauthors{1
Pascal Mettes
Dennis C. Koelma
Cees G.~M. Snoek
\affaddr{University of Amsterdam
This paper strives for video event detection using a representation learned from deep convolutional neural networks. Different from the leading approaches, who all learn from the 1,000 classes defined in the ImageNet Large Scale Visual Recognition Challenge, we investigate how to leverage the complete ImageNet hierarchy for pre-training deep networks. To deal with the problems of over-specific classes and classes with few images, we introduce a bottom-up and top-down approach for reorganization of the ImageNet hierarchy based on all its 21,814 classes and more than 14 million images. Experiments on the TRECVID Multimedia Event Detection 2013 and 2015 datasets show that video representations derived from the layers of a deep neural network pre-trained with our reorganized hierarchy i) improves over standard pre-training, ii) is complementary among different reorganizations, iii) maintains the benefits of fusion with other modalities, and iv) leads to state-of-the-art event detection results. The reorganized hierarchies and their derived Caffe models are publicly available at %%% ONLINE VERSION %\url{http://www.anonymized.com. %\url{https://staff.fnwi.uva.nl/p.s.m.mettes/projects/imagenetshuffle.html. \url{http://tinyurl.com/imagenetshuffle.
abstract.tex
\begin{abstract}
This paper strives for video event detection using a representation learned from deep convolutional neural networks. Different from the leading approaches, who all learn from the 1,000 classes defined in the ImageNet Large Scale Visual Recognition Challenge, we investigate how to leverage the complete ImageNet hierarchy for pre-training deep networks. To deal with the problems of over-specific classes and classes with few images, we introduce a bottom-up and top-down approach for reorganization of the ImageNet hierarchy based on all its 21,814 classes and more than 14 million images. Experiments on the TRECVID Multimedia Event Detection 2013 and 2015 datasets show that video representations derived from the layers of a deep neural network pre-trained with our reorganized hierarchy \textit{i)} improves over standard pre-training, \textit{ii)} is complementary among different reorganizations, \textit{iii)} maintains the benefits of fusion with other modalities, and \textit{iv)} leads to state-of-the-art event detection results. The reorganized hierarchies and their derived Caffe models are publicly available at
%%% ONLINE VERSION
%\url{http://www.anonymized.com}.
%\url{https://staff.fnwi.uva.nl/p.s.m.mettes/projects/imagenetshuffle.html}.
\url{http://tinyurl.com/imagenetshuffle}.
\end{abstract}

conclusions.tex
\section{Conclusions}
In this work, we leverage the complete ImageNet dataset for pre-training deep convolutional neural networks for video event detection, rather than the prescribed 1,000 ImageNet subset. We propose two contrasting and complementary approaches to reorganize the ImageNet hierarchy. The bottom-up approach aims to merge classes from the deepest parts of hierarchy upwards, while the top-down approach aims to select rich generic classes starting from the top of the hierarchy. The new hierarchies are in turn used as input to pre-train deep networks and are employed for frame representation in video event detection. Experimental evaluation performed on the challenging TRECVID MED 2013 dataset shows that deep networks trained on our hierarchies \textit{i)} outperform standard pre-trained networks, \textit{ii)} are complementary, \textit{iii)} maintain the benefits of fusion with other modalities, and \textit{iv)} reach state-of-the-art result. The pre-trained models are available online at
%%% ONLINE VERSION
%\url{http://www.anonymized.com}.
\url{http://tinyurl.com/imagenetshuffle}
and can be used directly to extract state-of-the-art video representations using the Caffe library.

experiments.tex
% FIGURE EXP 1
\begin{figure*}[t]
\centering
\begin{subfigure}{0.49\textwidth}
\includegraphics[width=\textwidth]{images/exp/exp1-bottomup-100ex.pdf}
\caption{TRECVID MED 2013 with 100 positives.}
\end{subfigure}
\begin{subfigure}{0.49\textwidth}
\includegraphics[width=\textwidth]{images/exp/exp1-bottomup-10ex.pdf}
\caption{TRECVID MED 2013 with 10 positives.}
\end{subfigure}
\caption{Mean Average Precision scores for the three bottom-up variants on TRECVID Multimedia Event Detection 2013. We observe that the more classes are maintained in the bottom-up reorganization, the better the performance using the soft-max (i.e. semantic) layer. The reserve happens for the fully-connected layer.}
\label{fig:exp1-bottomup}
\end{figure*}

% SECTION 4
\section{Experimental setup}

% SECTION 4.1
\subsection{Dataset}
\noindent
\textbf{TRECVID Multimedia Event Detection 2013.} The TRECVID Multimedia Event Detection 2013 dataset consists of roughly 27,000 test videos~\cite{over14}. The dataset contains annotations for 20 everyday events, including \emph{Birthday Party}, \emph{Making a sandwich}, \emph{Attempting a bike trick}, and \emph{Dog show}. The dataset has two different tasks, one where 10 positive videos are given for each event (10 Ex.), and one where 100 positive videos are given for each event (100 Ex.). For an event, a classifier is trained on the 10 or 100 positive videos and a background set of roughly 5,000 negative videos. The classifier is in turn used to rank the 27,000 test videos and its performance is evaluated using the (mean) Average Precision score on the ranked test videos.

% SECTION 4.2
\subsection{Implementation details}

\textbf{Deep convolutional networks.} We focus our evaluation on the recent GoogLeNet of Szegedy \etal~\cite{szegedy2014going}. The GoogLeNet is a deep convolutional neural network consisting of 22 layers. We also compare against the AlexNet of Krizhevsky \etal~\cite{krizhevsky2012imagenet}. The AlexNet consists of 5 convolutional layers and 3 fully-connected layers. To pre-train the deep networks, we utilize the open-source Caffe library~\cite{jia2014caffe} and the provided layer definitions and hyper-parameters for both networks.

\textbf{Feature extraction.} After pre-training, we extract features both at the fully connected layer and the soft-max layer. In AlexNet, we use the features from the second fully-connected layer, with a 4,096-dimensional frame representation. In GoogLeNet, we use the features at the pool5 layer, with a 1,024-dimensional frame representation. The dimensionality at the soft-max layer, which provides a probability score of each concept, for both networks is equal to the number of classes in the corresponding hierarchy.

\textbf{Pooling and Event Classification.} For event detection, we average the representations of the frames over each video unless stated otherwise, followed by $\ell_{1}$-normalization. We train an SVM classifier for each event separately with a $\chi_{2}$ kernel. We set the $C$ parameter to 100 in all our experiments.

% SECTION 5
\section{Experiments}
We consider four experiments. First, we evaluate the effect of different settings of the operations in our bottom-up reorganized pre-training. Second, we compare standard pre-training versus both the bottom-up and top-down reorganized pre-training. Third, we perform various fusions between deep representations and representation from other modalities. Fourth, we compare our results to the state-of-the-art on multimedia event detection.

% SECTION 5.1
\subsection{Bottom-up operation parameters}

\textbf{Experiment 1.} For the first experiment, we investigate the parameters for the bind and promote operations in our bottom-up reorganization, which have a significant influence on the amount of remaining classes. In total, we have trained three separate GoogLeNets~\cite{szegedy2014going} based on different parameters for the bind and promote operations:
\begin{itemize}
\item \textbf{Bottom-up [4k]:} Deep network pre-trained on 4,437 classes with $t_b=7,000$ and $t_p=1,250$.
\item \textbf{Bottom-up [8k]:} Deep network pre-trained on 8,201 classes with $t_b=7,000$ and $t_p=500$.
\item \textbf{Bottom-up [13k]:} Deep network pre-trained on 12,988 classes with $t_b=3,000$ and $t_p=200$.
\end{itemize}
For all the variants, we set the subsample threshold to $t_s=2,000$. An overview of mean Average Precision scores using the fully-connected and soft-max layers on TRECVID Multimedia Event Detection 2013 is shown in Figure~\ref{fig:exp1-bottomup}. We report the mean Average Precision scores both for the task with 10 positive videos and with 100 positive videos per event.

\textbf{Results.} From Figure~\ref{fig:exp1-bottomup}, we observe that the best scores using the fully-connected layer are achieved with the bottom-up [4k] variant. This result shows that the fully-connected layer translates best to events when merging more classes into a generic hierarchy. Interestingly, we observe the reverse pattern for the soft-max layer; the more classes are maintained the better the event detection performance. This result follows the work of Habibian et al.~\cite{habibian13}, which states that using more semantic classifiers is preferred over using better semantic classifiers. Here, we show that this observation translates to deep networks for event detection.

From this experiment, we conclude that the choice of the bottom-up reorganized variant depends on the desired deep network representation. The highest overall results are achieved by the features from the non-semantic fully-connected layer of the variant from 4,437 classes (0.446 mean Average Precision using 100 positives per event, 0.296 using 10 positives). However, the  variant from 12,988 classes performs best using the semantic features from the soft-max layer (0.441 using 100 positives, 0.286 using 10 positives).

% FIGURE EXP 2
\begin{figure*}[t]
\centering
\begin{subfigure}{0.49\textwidth}
\includegraphics[width=\textwidth]{images/exp/exp2-100ex.pdf}
\caption{TRECVID MED 2013 with 100 positives.}
\end{subfigure}
\begin{subfigure}{0.49\textwidth}
\includegraphics[width=\textwidth]{images/exp/exp2-10ex.pdf}
\caption{TRECVID MED 2013 with 10 positives.}
\end{subfigure}
\caption{Mean Average Precision scores for our bottom-up and top-down reorganized pre-training (blue), compared to standard pre-training (green) on TRECVID MED 2013. Our approaches both clearly outperform standard pre-training, while being competitive and potentially complementary to each other.}
\label{fig:exp2-compare}
\end{figure*}

% SECTION 5.2
\subsection{Standard versus reorganized pre-training}
\textbf{Experiment 2.} For the second experiment, we compare our bottom-up and top-down reorganized pre-training against the conventional pre-training setup using the ImageNet 1,000 class subset~\cite{russakovsky2014imagenet}. For all networks, we report the Average Precision scores using both the fully-connected layer and the soft-max layer. For the bottom-up approach, we use the deep network pre-trained on 4,437 classes. For the top-down approach, we select the top 4,000 classes, with $t_t = 1,200$ for the threshold on the number of images required for each class. We compare our two approaches to two standard pre-trained deep networks:
\begin{itemize}
\item \textbf{AlexNet [std]:} AlexNet pre-trained on 1,000 ImageNet classes~\cite{krizhevsky2012imagenet}.
\item \textbf{GoogLeNet [std]:} GoogLeNet pre-trained on 1,000 ImageNet classes~\cite{szegedy2014going}.
\end{itemize}


\textbf{Results.} An overview of the comparison between standard and reorganized pre-training is shown in Figure~\ref{fig:exp2-compare}. We observe that the top-down and bottom-up reorganization approaches achieve comparable performance. While bottom-up performs slightly better using the fully-connected layer, top-down performs slightly better using the soft-max layer. We also note our reorganized pre-training approaches on GoogLeNet significantly outperform the standard pre-trained GoogLeNet. This holds especially for the soft-max layer, where the difference between standard pre-training and our top-down pre-training is 8.6\% and 9.2\% in absolute mean Average Precision for respectively the 100 and 10 positive video tasks. Lastly, we note that the difference in performance to the pre-trained AlexNet is even bigger. This result shows that GoogLeNet provides overall better visual representations, leading to improved event detection.

From this experiment, we conclude that our two approaches to reorganized ImageNet pre-training yield strong event detection results and significantly improve over standard pre-trained deep networks.

\subsection{Fusing representations and modalities}

\textbf{Experiment 3.} For the third experiment, we investigate the effect of feature fusion. Here, fusion is performed in a late fashion, by averaging the classifier scores of different classifiers. We investigate feature fusion in two aspects: \textit{i)} we investigate the effect of fusing different layers and video encodings from different pre-trained deep networks for event detection, \textit{ii)} we investigate the effect of fusing our deep visual representations with two other representations:
\begin{itemize}
\item \textbf{Audio modality:} MFCC features with first and second order derivatives, 30 dimensions for each of the three features, aggregated into a 46,080-dimensional video representation using Fisher Vectors with 256 clusters.
\item \textbf{Motion modality:} MBHx, MBHy, and HOG features computed along dense trajectories~\cite{wang13}, reduced to 128 dimensions using PCA, aggregated into a 65,536-dimensional video representation using Fisher Vectors with 256 clusters.
\end{itemize}

% SECTION 5.3
\textbf{Results for Fusing Networks.} %{Fusing deep network layers and networks}
In Table~\ref{tab:fusedeep}, we show an overview of fusion results using deep networks. Comparing index (1) to (3) and comparing index (2) to (4), we see that for both the bottom-up and top-down approach, it is beneficial to fuse the scores from the fully-connected and soft-max layers. This result is surprising, given that the layers come from the same network and it indicates that the layers contain different information useful for event detection. This result is furthermore interesting from a computational perspective, as the features from both layers can be extracted from a single pass through the same network. Hence, the improvement is obtained for free.

The fusion of (3) and (4), i.e. the fusion of the bottom-up and top-down reorganizations, also yields complementary results, with a mean Average Precision of 0.475 and 0.324 for respectively the 100 and 10 positive video tasks of the TRECVID Multimedia Event Detection 2013 dataset. This result clearly shows that pre-training deep networks on different hierarchies results in different and complementary representations. Figure~\ref{fig:deepfuseperevent} shows that, although the mean Average Precision of the individual approaches is similar, the scores per event vary notably (on average 2.7\% per event), resulting in improved performance upon fusion.

% TABLE 1 EXP 3
\begin{table}[t]
\centering
%\small
\begin{tabular}{l r r}
\toprule
\multicolumn{3}{c}{\textbf{TRECVID MED 2013}}\\
Method & 100 Ex. & 10 Ex.\\
\midrule
\textbf{Averaging} & &\\
(1) Bottom-up (fc) & 0.446 & 0.296\\
(2) Top-down (fc) & 0.438 & 0.300\\
(3) Bottom-up (fc + soft-max) & 0.452 & 0.305\\
(4) Top-down (fc + soft-max) & 0.454 & 0.317\\
(3) + (4) & 0.475 & 0.324\\
\midrule
\textbf{VLAD} & &\\
Bottom-up (fc + soft-max) & 0.465 & 0.339\\
\bottomrule
\end{tabular}
\caption{Mean Average Precision scores for fusions of different layers, networks, and encodings within deep representations, which all yield complementary results.}
\label{tab:fusedeep}
\end{table}

Multiple recent works have investigated complex and high-dimensional video representations from deep frame representations beyond frame averaging~\cite{nagel2015event,xu2014discriminative}. Here, we similarly investigate such representations using the frame features with reorganized pre-training. We have employed both VLAD and Fisher Vector encoding and report the results for VLAD in Table~\ref{tab:fusedeep}, as that yielded the highest scores. For the VLAD encoding, we create a codebook from 10 clusters per event, resulting in a 10,240-dimensional feature vector using the fully-connected layer and a 44,370-dimensional feature vector using the soft-max layer. The results using the bottom-up reorganization show that a VLAD encoding improves over averaging, especially for the 10 positive videos per event task (0.339 mAP versus 0.305 for averaging).

We conclude from this fusion experiment that combining information from different pre-trained deep networks and even different layers from the same deep network improves the Average Precision scores. Furthermore, performing a VLAD encoding instead of averaging frames results in a boost for individual networks, especially for the 10 positive videos task.

% FIGURE EXP 3
\begin{figure}[t]
\includegraphics[width=\linewidth]{images/exp3-deepfusion-perevent.pdf}
\caption{The Average Precision scores per event for bottom-up, top-down, and their fusion. Note that the scores per event are different for the two approaches, resulting in complementary fusion results.}
\label{fig:deepfuseperevent}
\end{figure}

% SECTION 5.4
\textbf{Results for Fusing Modalities.} %{Fusing with other modalities}
In Table~\ref{tab:fuseother}, we show the results of the deep networks with the audio and motion modalities. The Table clearly states that individually, the event detection scores using our deep networks improve over the motion and audio scores. Upon combining the modalities, we observe a jump in performance. This result shows the complementary natures of the different modalities: individually the motion and audio features are clearly outperformed, but they contain information not captured in deep networks which result in improved fusion results. This is naturally due to the nature of deep convolutional neural networks, which focus on spatially visual information and exclude temporal and audio information.

We have furthermore attempted to fuse the VLAD encoding of Table~\ref{tab:fusedeep} with the motion and audio features, but this did not result in improved performance. Since the VLAD encoding requires more computational effort and has a higher storage requirement, we have opted to focus on averaging.

\subsection{Comparison to the state-of-the-art}
\textbf{Experiment 4.} For the fourth experiment, we compare our results to the current state-of-the-art on multimedia event detection. We perform a comparison on both the TRECVID MED 2013 test set and the TRECVID MED 2015 benchmark.

\textbf{Results on the TRECVID MED 2013 Test set.} The comparison to the state-of-the-art on the TRECVID Multimedia Event Detection 2013 dataset is shown in Table~\ref{tab:sota2013}. As the Table shows, we outperform the current state-of-the-art on both the 100 and 10 positive videos per event task using deep networks only. Upon a fusion with motion and audio features, we improve further over related work.

% TABLE 2 EXP 3
\begin{table}[t]
\centering
%\small
\begin{tabular}{l r r}
\toprule
\multicolumn{3}{c}{\textbf{TRECVID MED 2013}}\\
Method & 100 Ex. & 10 Ex.\\
\midrule
\textbf{Individual} & &\\
(1) Audio (MFCC) & 0.114 & 0.053\\
(2) Motion (MBH) & 0.341 & 0.192\\
(3) Visual (ours, avg) & 0.475 & 0.324\\
\midrule
\textbf{Fusion} & &\\
(2) + (3) & 0.504 & 0.345\\
(1) + (2) + (3) & 0.526 & 0.348\\
\bottomrule
\end{tabular}
\caption{Mean Average Precision scores for fusing our deep representations with other modalities. We outperform motion and audio features, while the fusion leads to further improvement.}
\label{tab:fuseother}
\end{table}

\begin{table}[t]
\centering
%\scriptsize
\begin{tabular}{l r r}
\toprule
\multicolumn{3}{c}{\textbf{TRECVID MED 2013}}\\
Method & 100 Ex & 10 Ex\\
\midrule
Habibian et al. \cite{habibian14} 		& - 	& 0.196\\
Sun et al. (visual) \cite{sun142} 	    & 0.350 & -\\
Nagel et al. \cite{nagel2015event} 		& 0.386 & 0.218\\
Sun et al. (fusion) \cite{sun142}        & 0.425 & -\\
Xu et al. \cite{xu2014discriminative} 	& 0.446 & 0.298\\
Chang et al. \cite{chang2015searching} 	& - 	& 0.310\\
\midrule
Ours, deep network 						& 0.475 & 0.324\\
Ours, multimodal fusion 				& 0.526 & 0.348\\
\bottomrule
\end{tabular}
\caption{Comparison to other works on TRECVID MED 2013 test set for both our best deep network results and our fusion results. We yield better results for both the 100 and 10 positive videos per event task.}
\label{tab:sota2013}
\end{table}

% FIGURE EXP 4
%\begin{figure*}[t]
%\includegraphics[width=\textwidth]{images/exp/med2015-allres.pdf}
%\caption{Comparison between our results (blue) and the results of all other participants in the TRECVID Multimedia Event Detection 2015. Our deep networks with motion and audio fusion is the top contender for all tasks, while our deep networks by themselves are readily outperforming nearly all other participants.}
%\label{fig:sota2015}
%\end{figure*}
\begin{figure*}[t]
\centering
\begin{subfigure}{0.32\textwidth}
\includegraphics[width=\textwidth]{images/exp4-2015-10ah.pdf}
\end{subfigure}
\begin{subfigure}{0.32\textwidth}
\includegraphics[width=\textwidth]{images/exp4-2015-10ps.pdf}
\end{subfigure}
\begin{subfigure}{0.32\textwidth}
\includegraphics[width=\textwidth]{images/exp4-2015-100ps.pdf}
\end{subfigure}
\caption{Comparison between our results (blue) and the results of all other participants in the TRECVID Multimedia Event Detection benchmark 2015. Our deep networks and their fusion with motion and audio information are the top contenders for all tasks.}
\label{fig:sota2015}
\end{figure*}

\textbf{Results on the TRECVID MED 2015 Benchmark.} We furthermore compare our results achieved on the latest TRECVID 2015 benchmark for Multimedia Event Detection. This benchmark is similar in nature to the 2013 dataset in training and evaluation. However, the 2015 dataset contains 20 new events. Furthermore, the benchmark comparison is performed in a large-scale setup, with a test set of about 200,000 videos. In Figure~\ref{fig:sota2015}, we show the inferred mean Average Precision scores for our entries and the entries of the other participants. We report results both for the pre-specified (where the events and video labels are given well before the benchmark deadline) and ad-hoc (where the events and video labels are given shortly before the benchmark deadline) tasks. The Figure paints a similar picture to the results on the 2013 dataset; we outperform the current state-of-the-art by fusing deep representations with motion and audio modalities, while our deep representations only are already among the top contenders.

hierarchy.tex
\section{Reorganized pre-training}

\begin{figure}[t]
\includegraphics[width=\linewidth]{images/hierarchy.pdf}
\caption{Overview of where in the hierarchy the four operations are applied in the bottom-up approach. \textbf{Roll:} Roll up classes with a single chld-parent connection. \textbf{Bind:} Bind sub-trees for which the individual classes do not have enough images. \textbf{Promote:} Promote individual classes to their parent class if they do not have enough images. \textbf{Subsample:} Randomly subsample images from classes with too many images.}
\label{fig:reorganize}
\end{figure}

\begin{figure*}[t]
\centering
\begin{subfigure}{\textwidth}
\centering
\includegraphics[width=0.275\textwidth]{images/operations/roll-1.pdf}
\hspace{0.5cm}
\includegraphics[width=0.275\textwidth]{images/operations/roll-2.pdf}
\hspace{0.5cm}
\includegraphics[width=0.275\textwidth]{images/operations/roll-3.pdf}
\caption{Roll.}
\label{fig:roll}
\end{subfigure}
\noindent\rule[0.5ex]{\linewidth}{0.75pt}
\vspace{0.015cm}\\
\begin{subfigure}{\textwidth}
\centering
\includegraphics[width=0.27\textwidth]{images/operations/bind-1.pdf}
\hspace{1.0cm}
\includegraphics[width=0.27\textwidth]{images/operations/bind-2.pdf}
\hspace{1.0cm}
\includegraphics[width=0.27\textwidth]{images/operations/bind-3.pdf}
\caption{Bind.}
\label{fig:bind}
\end{subfigure}
\noindent\rule[0.5ex]{\linewidth}{0.75pt}
\vspace{0.015cm}\\
\begin{subfigure}{\textwidth}
\centering
\includegraphics[width=0.2\textwidth]{images/operations/promote-1.pdf}
\hspace{1.5cm}
\includegraphics[width=0.2\textwidth]{images/operations/promote-2.pdf}
\hspace{1.5cm}
\includegraphics[width=0.2\textwidth]{images/operations/promote-3.pdf}
\caption{Promote.}
\label{fig:promote}
\end{subfigure}
\noindent\rule[0.5ex]{\linewidth}{0.75pt}
\vspace{0.015cm}\\
\begin{subfigure}{\textwidth}
\centering
\includegraphics[width=0.745\textwidth]{images/operations/subsample.pdf}
\caption{Subsample.}
\label{fig:subsample}
\end{subfigure}
\caption{Visual examples for each of the four operations for the bottom-up reorganization.}
\label{fig:operations}
\end{figure*}

%The classes in the ImageNet dataset are a subset of the WordNet collection~\cite{miller1995wordnet} and the classes are therefore connected in a hierarchy. The connectivity between classes provides information about the semantic relationship between classes. We utilize the hierarchical relationship of WordNet for combining classes to generate a new balanced and generalized ImageNet hierarchy. We start from the original ImageNet hierarchy and introduce four operations to reorganize it. An overview of where in the hierarchy the four operations are performed is shown in Figure~\ref{fig:reorganize} and visual examples for each operation is shown in Figure~\ref{fig:operations}. We outline each operation separately below.
The classes in the ImageNet dataset are a subset of the WordNet collection~\cite{miller1995wordnet} and the classes are therefore connected in a hierarchy. The connectivity between classes provides information about their semantic relationship. We utilize the hierarchical relationship of WordNet for combining classes to generate reorganized ImageNet hierarchies for pre-training. We focus on two opposing approaches for reorganization, namely a bottom-up and top-down approach.

\subsection{Bottom-up reorganization}
For the bottom-up reorganization, we start from the original ImageNet hierarchy and introduce four reorganization operations. An overview of where in the hierarchy the four operations are performed is shown in Figure~\ref{fig:reorganize} and visual examples for each operation are shown in Figure~\ref{fig:operations}. We outline each operation separately.
\\
\textbf{Roll.} The roll operation is performed on single-link sub-trees of classes. In other words, the roll operation merges classes with a single child-parent connection, as shown in Figure~\ref{fig:reorganize} on the left. The motivation behind this operation is two-fold: \textit{i)} there is little semantic difference between a child and a parent if the parent has no other children. Treating the child and parent as separate classes during pre-training will dominate the backpropagation gradients to keep these classes separated. \textit{ii)} A single child of a parent is more likely to be over-specific for event detection. Single child-parent connections typically occur deep in the hierarchy, where details between classes become more fine-grained. In our evaluation, we indeed observe that the single child-parent connections occur predominantly in the deeper layers of the ImageNet hierarchy. Three chains of single child-parent connections are shown in Figure~\ref{fig:roll}. For example, the class \emph{Mamba} is a type of snake and has a single child, namely \emph{Black mamba}. In turn, the \emph{Black mamba} has a single child: \emph{Green mamba} (the green phase of the black mamba). In this example, we move all the images from the \emph{Black mamba} and \emph{Green mamba} classes to the \emph{Mamba} class.
% TABLE ROLL
%\begin{table}[t]
%\centering
%\scriptsize
%\begin{tabular}{l l l l}
%\toprule
%\multicolumn{4}{c}{\textbf{Roll}} \\
%\midrule
%Leopard fur & Fur & Animal skin & Animal product\\
% & Green mamba & Black mamba & Mamba\\
% & Nacho & Tortilla chip & Corn chip\\ 
% & Bucking bronco & Bronce & Mustang\\
%\bottomrule
%\end{tabular}
%\caption{Four examples of the roll operation. For each row, the images of the classes have iteratively moved up from left to right.}
%\label{tab:roll}
%%\end{table}
%\vspace{0.5cm}
%% TABLE BIND
%%\begin{table}[t]
%%\centering
%%\scriptsize
%\begin{tabular}{l l l | l}
%\toprule
%\multicolumn{4}{c}{\textbf{Bind}} \\
%%\midrule
%\multicolumn{3}{c}{Children} & \multicolumn{1}{c}{Parent} \\
%\midrule
%Zeppelin & Big air balloon & Trial balloon & Balloon\\
%Smalleye & Smooth hammerhead & Shovelhead & Hammerhead\\
% & African clawed frog & Surinam toad & Tongueless frog\\
%\bottomrule
%\end{tabular}
%\caption{Three examples of the bind operation ($t_b < 3,000$). The children and parent are merged into a single class.}
%\label{tab:bind}
%%\end{table}
%\vspace{0.5cm}
%% TABLE PROMOTE
%%\begin{table}[t]
%%\centering
%%\scriptsize
%\begin{minipage}{.6\linewidth}
%\centering
%\begin{tabular}{l l}
%\toprule
%\multicolumn{2}{c}{\textbf{Promote}} \\
%%\midrule
%\multicolumn{1}{c}{Child} & \multicolumn{1}{c}{Parent} \\
%\midrule
%New Yorker & American\\
%Triclinium & Dining table\\
%Card table & Table\\
%Pleurodont & Animal\\
%Anchovy sauce & Sauce\\
%Wall & Object\\
%\bottomrule
%\end{tabular}
%\caption{Four examples of the promote operation ($t_p < 200$), where the images of the child are added to the parent class.}
%\label{tab:promote}
%\end{minipage}
%\hfill
%% TABLE SUBSAMPLE
%\begin{minipage}{.3\linewidth}
%\centering
%\begin{tabular}{l}
%\toprule
%\multicolumn{1}{c}{\textbf{Subsample}} \\
%\midrule
%Sauce\\
%Table\\
%Person\\
%Herb\\
%Coffee mug\\
%Butterfly\\
%Flame flower\\
%\bottomrule
%\end{tabular}
%\caption{Five examples of the subsample operation ($t_s > 2000$).}
%\label{tab:subsample}
%\end{minipage}
%\end{table}
\\
\textbf{Bind.} The bind operation is performed on sub-trees where the individual classes are sparse in the number of images. Let $S$ denote a sub-tree and let $c^i$ denote the number of images in class $c$. Then the bind operation is performed on sub-tree $S$ if $\sum_{c \in S} c^i < t_b$, where $t_b$ denotes the threshold on the number of images. The notion behind the bind operation is to deal with small and semantically coherent classes consisting of a parent and multiple children. The children individually do not contain enough images to treat them as separate classes. However, the combined set of parent and children forms a semantically consistent set with a desirable number of images. Three merged sub-trees that are combined with the bind operation are shown in Figure~\ref{fig:bind}. For example, the \emph{Hammerhead shark} has three children with a small number of images, namely \emph{Smooth hammerhead}, \emph{Smalleye hammerhead}, and \emph{Shovelhead}. Therefore, we opt to combine all these shark images into a single class.
\\\\
\textbf{Promote.} The promote operation is a unary operation. It is performed after the roll and bind have been performed. The promote operation simply promotes a class to its parent if its number of images is below a threshold $t_p$. This operation directly targets the imbalance problem, by adding images of classes with few examples to parent classes with more images. Figure~\ref{fig:promote} shows three cases of the promote operation. For example, the class \emph{Triclinium} (a dining table with couches at three sides) only contains 5 images. Therefore, the images are added to the \emph{Dining table} class, such that the \emph{Triclinium} images are still being used for pre-training without creating an imbalance in the hierarchy.
\\\\
\textbf{Subsample.} The subsample operation is also a unary operation and deals with the reverse problem of the other three operations. The subsample operation subsamples images from classes for which the number of images is above a threshold $t_s$. The reason for this operation is again for balancing purposes. If all images of over-populated classes are used in the stochastic optimization of the deep network, the network will overfit to these classes, resulting in suboptimal frame representations for event detection. Four examples of the subsample operation are shown in Figure~\ref{fig:subsample}, such as \emph{Keyboard}, \emph{Coffee mug}, and \emph{Herb}.
\\\\
We employ the defined operations in the described order. First, all single child-parent connections are rolled up. Second, all sub-trees in the hierarchy are binded based on threshold $t_b$ for their combined number of images. Third, all remaining classes with less than $t_p$ images are promoted to their parent. Fourth, during network pre-training, examples for all classes with more than $t_s$ images are randomly sub-sampled before the stochastic gradient descent optimization.

%\begin{figure}[t]
%\centering
%\includegraphics[width=0.95\linewidth]{images/topdown.pdf}
%\caption{Overview of the top-down reorganization, where at a new layer $l+1$, the corresponding classes are first sorted by number of images (indicated by the color of the blobs). Then, we iteratively added the classes with enough images before moving to the new layer in a breath-first search manner.}
%\label{fig:topdown}
%\end{figure}


\subsection{Top-down reorganization}
An alternative and potentially complementary reorganization strategy is not to start from the deepest classes in the hierarchy, but from the head node. Here, we investigate a breath-first search approach. Let $t_t$ denotes a pre-specified threshold stating the minimum amount of images required for class in order to be used in the top-down reorganization. Then, starting from layer 0 in the hierarchy, i.e. the head node, we iteratively move down in the hierarchy and keep adding classes with at least $t_t$ images until we reach a desired amount of classes.

The breath-first search approach is outlined as follows. Let $l$ denote the previous layer of the hierarchy. We list all ImageNet classes in layer $l+1$ based on connections from classes in layer $l$ and order the classes in $l+1$ by their number of images. The sorting ensures that we select the classes with the highest number of images first, in case we reach the desired amount of classes before the end of the list. Then, we move through the ordered list and select all classes with at least $t_t$ images as long as the desired amount of selected classes is not reached. Afterwards, we move to the next layer and repeat the ordering and selection procedure.

By using a top-down approach, we ensure that only the most general classes are maintained for pre-training, while simultaneously keeping a balance in the image distribution through the threshold $t_t$.

introduction.tex
\section{Introduction}
%This work aims at optimizing the pre-training of deep convolutional neural networks for video event detection and semantic interpretation. A large body of work has shown that deep networks are effective for events. This is most notably due to their discriminative power~\cite{mettes2015bag,nagel2015event} and their semantic representations, which enable zero-shot event detection~\cite{cappallo2015image,jain2015objects} and event recounting~\cite{mettes2015bag,sun142,sun14}. To pre-train the deep networks, large-scale image collections are employed. We focus on how to best leverage the image collections during network pre-training.

The goal of this work is to detect events such as \emph{Renovating a home}, \emph{Birthday party}, and \emph{Attempting a bike trick} in web videos. The leading approaches~\cite{jiang2015fast,mettes2015bag,nagel2015event,xu2014discriminative} attack this challenging problem by learning video representations through a deep convolutional neural network~\cite{krizhevsky2012imagenet,szegedy2014going}. The deep network is pre-trained on a collection of 1,000 ImageNet classes~\cite{imagenet09,russakovsky2014imagenet}, used to  extract features for video frames, and then followed by a pooling operation over the frames to arrive at a video representation. We also learn representations for event detection with a deep convolutional neural network, but rather than relying on the default 1,000 class subset, we investigate how to leverage the complete ImageNet hierarchy for pre-training the representation. 

\begin{figure}[t]
\begin{subfigure}{\linewidth}
\includegraphics[width=\linewidth]{images/concept-counts-fig1.pdf}
\caption{Distribution of the number of images for the 21,814 ImageNet classes. Note the class imbalance.}
\label{fig:imbalance}
\end{subfigure}
\begin{subfigure}{\linewidth}
\centering
\includegraphics[width=0.49\linewidth]{images/siderocyte.png}
\includegraphics[width=0.49\linewidth]{images/gametophyte.png}
\caption{An image of \emph{Siderocyte} (left) and \emph{Gametophyte} (right), two classes which seem over-specific for event detection.}
\label{fig:overspecific}
\end{subfigure}
\caption{Two problems when using the full ImageNet hierarchy for network pre-training: (a) image imbalance and (b) over-specific classes. In this work, we aim to reorganize the hierarchy into a balanced set of classes for more effective pre-training of video representations for event detection.}
\label{fig:problems}
\end{figure}

The complete ImageNet dataset consists of over 14 million images and 21,814 classes which are connected in a hierarchy as a subset of WordNet~\cite{miller1995wordnet}. State-of-the-art event detectors are pre-trained on a 1,000 class (1.2 million images) subset of ImageNet, as prescribed by the Large Scale Visual Recognition Challenge~\cite{russakovsky2014imagenet}. Hence, more than 90\% of the images in the hierarchy remain untouched during pre-training. We present an empirical investigation of the effect of using the full ImageNet dataset for event detection in web videos.

We identify two problems when trying to pre-train a deep network on the complete ImageNet hierarchy. First, there is an imbalance in the number of examples for each class, as is shown in Figure~\ref{fig:imbalance}. For example, the class \emph{Yorkshire Terrier} contains 3,072 images, whereas 296 other classes contain just a single image. Second, some classes seem over-specific for event detection in web videos. Consider for example the ImageNet categories \emph{Siderocyte} and \emph{Gametophyte} in Figure~\ref{fig:overspecific}. As a result, it seems suboptimal to directly pre-train a deep network on all 21,814 classes.

In this work, we introduce pre-training protocols that reorganize the full ImageNet hierarchy for more effective pre-training. The reorganization tackles the image imbalance and over-specific class problems. We propose two contrasting approaches that utilize the graph structure of ImageNet to combine and merge classes into balanced and reorganized hierarchies. We empirically evaluate our event detection using reorganized pre-training on the 2013 and 2015 NIST TRECVID Multimedia Event Detection datasets, for both datasets leading to state-of-the-art results. The Caffe models and detailed video feature extraction instructions are available at
%%% ONLINE VERSION
%\url{http://www.anonymized.com}.
\url{http://tinyurl.com/imagenetshuffle}.

%The de facto standard image collection for network pre-training is the ImageNet dataset~\cite{imagenet09}. The complete ImageNet dataset consists of over 14 million imagse and 21,814 classes which are connected in a hierarchy. However, deep networks are generally pre-trained on a 1,000 class (1.2 million images) subset of ImageNet, as prescribed by the Large Scale Visual Recognition Challenge~\cite{russakovsky2014imagenet}. This means that over 90\% of the images in the hierarchy generally remain unused for pre-training. Here, we adress the effect of employing the full ImageNet dataset for detecting and recounting events.

%We identify two problems when trying to pre-train a deep network on the complete ImageNet hierarchy. First, there is an imbalance in the number of examples for each class, as is shown in Figure~\ref{fig:imbalance}. For example, the class \emph{Yorkshire Terrier} contains 3,072 images, while on the other hand, 296 classes contain a single image. Second, some classes are over-specific for event detection and recounting. Figure~\ref{fig:overspecific} shows shows the ImageNet categories \emph{Siderocyte} and \emph{Gametophyte}, both of which are intuitively over-specific for events as seperate classes. As a result, it is not optimal to directly pre-train a deep network on all 21,814 ImageNet classes.

%We propose to reorganize the ImageNet hierarchy to tackle the image imbalance and over-specific class problems. We introduce four operations aimed at merging nodes and sub-trees in the directed acyclic graph of the ImageNet hierarchy. The operations are aimed at grouping nodes and sub-trees of the ImageNet hierarchy based on the number of images they contain and based on their parent-child connections. Applying our operations results in a balanced hierarchy containing all images, without focusing on over-specific classes.

%The deep network trained on our hierarchy was the driving force behind the winning entries of the ad-hoc and pre-specified (10 Ex and 100 Ex) tasks of the latest TRECVID Multimedia Event Detection (MED) benchmark~\cite{snoek2015qualcomm}. Experimental evaluation in this paper on the TRECVID MED 2013 dataset shows that using deep networks pre-trained on our proposed hierarchy improves over (i) standard pre-trained deep networks, (ii) deep networks pre-trained directly on (nearly) all classes, and (iii) well-known shallow video representations. The Caffe models and detailed feature extraction instructions have been made available online at [...] to [...].

relatedwork.tex
\section{Related work}

\subsection{Event detection with pre-trained networks}

The state-of-the-art for event detection in videos focuses on video representations learned with the aid of deep convolutional neural networks~\cite{jiang2015fast,mettes2015bag,nagel2015event,xu2014discriminative}. The pipeline of these approaches consists roughly of three components. (1) A deep network is pre-trained on a large-scale image collection. Different deep networks have been employed for event detection, such as AlexNet~\cite{krizhevsky2012imagenet} in~\cite{jiang2015fast,mettes2015bag,nagel2015event} and VGGnet~\cite{simonyan2014very} in~\cite{xu2014discriminative}. (2)  Sampled video frames are fed to the network and features at fully-connected and/or soft-max layers are used as frame representations. (3) The frame representations are pooled into a fixed-sized video representation. A simple and effective pooling method is average pooling, where the frame representations are averaged over the video~\cite{karpathy2014large,mettes2015bag}. Recently, several works have shown that clustering deep frame representations into a codebook, followed by a VLAD~\cite{jegou2012aggregating} in~\cite{xu2014discriminative} or Fisher Vector encoding~\cite{sanchez2013image} in~\cite{nagel2015event}, leads to strong video representations. In this work, we aim for a similar pipeline of pre-training, frame representation, and video pooling. However, rather than relying on the standard pre-training protocol using 1,000 ImageNet classes, we leverage the complete ImageNet hierarchy for more effective pre-training.

Web videos provide a wide range of information about events, such as visual, motion, audio, and optical character information~\cite{over14}. Naturally, multiple works have investigated the fusion of information from different modalities~\cite{lan2014multimedia,myers2014evaluating,natarajan2012multimodal}. In this work, we also investigate the effect of fusing our deep representations with Motion Boundary Histogram (motion) features~\cite{wang13} and MFCC (audio) features~\cite{myers2014evaluating}, both of which are encoded into a video representation using Fisher Vectors~\cite{sanchez2013image}. This fusion allows us to compare the effectiveness of our deep representations to heterogeneous representations and to investigate how well our deep representations fare when combined with other sources of information.

\subsection{(Re)organizing hierarchies for events}

Various works have investigated the use of semantic hierarchies and ontologies for event detection. The work of Ye \etal~\cite{ye2015eventnet} focuses on hierarchical relations between events, to find a large collection of videos and event-specific concept classes. Their proposed EventNet has shown to yield an effective event detection~\cite{ye2015eventnet}. In our work, we focus on different hierarchical relations, namely between concept classes instead of events, to discover a general set of concepts for deep network pre-training. Other recent work on event detection has investigated relations among concept classes to rerank concept scores in the video representation~\cite{jiang2015fast}. We similarly focus on hierarchical relations among concept classes, but for the purpose of merging classes into a reorganized hierarchy for pre-training. For the hierarchy of ImageNet specifically, the work of Vreeswijk \etal~\cite{vreeswijk2012all} has shown that images from different layers of the hierarchy are visually different and that general concepts benefit from including linked concepts deeper in the hierarchy. We build upon these observations in our operations to reorganize the ImageNet hierarchy.

An alternative approach is to adjust concept hierarchies after feature extraction. For example, the selection of event-specific concepts based on the similarity to a textual event description has shown to yield effective event detection results without positive examples~\cite{jiang2015fast}. Mazloom \etal~\cite{mazloom2014conceptlets} show that concept selection is also beneficial for few-example event detection. Habibian \etal~\cite{habibian14} in turn, jointly learn a classifier for event detection and combine correlated concepts. Rather than changing the representations \emph{a posteriori} using text or video examples, we focus in this paper on reorganizing the hierarchical structure of visual ontologies \emph{before} event training, 


%A wide range of works exploit shallow low-level representations for event detection based on e.g. visual, motion, or audio modalities~\cite{myers2014evaluating,nagel2015event,natarajan2012multimodal}. For the visual modality, a popular approach is to first extract local image features such as SURF~\cite{bay2008speeded} or SIFT~\cite{lowe2004distinctive}. The local features are then aggregated into a VLAD~\cite{jegou2012aggregating} or Fisher Vector~\cite{nagel2015event,sanchez2013image} representation. Video representations based on motion compute local features at Spatio-Temporal Interest Points~\cite{everts14} or Dense Trajectories~\cite{wang13}, followed by a similar VLAD or Fisher Vector aggregation as the visual modality. Audio representations employ a similar representation using locally computed MFCC features~\cite{myers2014evaluating,natarajan2012multimodal}. Due to their low-level nature, results obtained by these methods are not interpretable. Furthermore, we show in this work that deep representations lead to improved event detection performance.

%Several recent works have investigated the use of semantic representations for events~\cite{habibian13,liu13,mazloom2014conceptlets,mazloom13,merler12}. The use of semantic representations allows for interpretable event results~\cite{sun142,sun14,tsai14}, as well as zero-shot event detection~\cite{mazloom2014few}. A semantic representation is typically obtained by training attribute classifiers on top of low-level representations~\cite{habibian13,mazloom13,merler12}. Each video is in turn represented by a vector of attribute scores. Other approaches to semantic representations include discovering an embedding between low-level representations and correlated semantic terms~\cite{habibian14} and propagating semantic tags from low-level visually neighbouring videos~\cite{mazloom2014few}. In this work, we similarly aim for event detection through semantic representations, but using deep convolutional network. This results both in faster feature extraction and in better event detection performance.

%The use of deep convolutional neural networks was first popularized due to their strong performance for large-scale image classification~\cite{krizhevsky2012imagenet,szegedy2014going}. Recent works have shown that deep networks pre-trained on large-scale image datasets are effective for other visual recognition tasks, such as action recognition~\cite{jain2015what} and multimedia event detection~\cite{mettes2015bag,nagel2015event}. A direct approach is to use available deep networks -- pre-trained on 1,000 ImageNet classes -- for recognizing actions and events~\cite{}. Several recent works have moved to using deep networks trained on 15,000 ImageNet classes (all classes with at least 200 images)~\cite{jain2015what,mettes2015bag}. The use of more semantic classes is in line with earlier work on semantic representations, which states that increasing the number of semantic classes benefits event detection~\cite{habibian13}. Here, we make the step towards fully leveraging the ImageNet dataset for pre-training deep networks. Our approach has three advantages over using the 15,000 class subset of ImageNet: 1) we exploit more images during pre-training, 2) we avoid overfitting to classes with a lot of examples and to over-specific classes, and 3) our semantic representation is more compact.

the-imagenet-shuffle-arxiv.tex
% This is "sig-alternate.tex" V2.0 May 2012
% This file should be compiled with V2.5 of "sig-alternate.cls" May 2012
%
% This example file demonstrates the use of the 'sig-alternate.cls'
% V2.5 LaTeX2e document class file. It is for those submitting
% articles to ACM Conference Proceedings WHO DO NOT WISH TO
% STRICTLY ADHERE TO THE SIGS (PUBS-BOARD-ENDORSED) STYLE.
% The 'sig-alternate.cls' file will produce a similar-looking,
% albeit, 'tighter' paper resulting in, invariably, fewer pages.
%
% ----------------------------------------------------------------------------------------------------------------
% This .tex file (and associated .cls V2.5) produces:
%       1) The Permission Statement
%       2) The Conference (location) Info information
%       3) The Copyright Line with ACM data
%       4) NO page numbers
%
% as against the acm_proc_article-sp.cls file which
% DOES NOT produce 1) thru' 3) above.
%
% Using 'sig-alternate.cls' you have control, however, from within
% the source .tex file, over both the CopyrightYear
% (defaulted to 200X) and the ACM Copyright Data
% (defaulted to X-XXXXX-XX-X/XX/XX).
% e.g.
% \CopyrightYear{2007} will cause 2007 to appear in the copyright line.
% \crdata{0-12345-67-8/90/12} will cause 0-12345-67-8/90/12 to appear in the copyright line.
%
% ---------------------------------------------------------------------------------------------------------------
% This .tex source is an example which *does* use
% the .bib file (from which the .bbl file % is produced).
% REMEMBER HOWEVER: After having produced the .bbl file,
% and prior to final submission, you *NEED* to 'insert'
% your .bbl file into your source .tex file so as to provide
% ONE 'self-contained' source file.
%
% ================= IF YOU HAVE QUESTIONS =======================
% Questions regarding the SIGS styles, SIGS policies and
% procedures, Conferences etc. should be sent to
% Adrienne Griscti (griscti@acm.org)
%
% Technical questions _only_ to
% Gerald Murray (murray@hq.acm.org)
% ===============================================================
%
% For tracking purposes - this is V2.0 - May 2012

\documentclass{sig-alternate-2013}

\usepackage{color}
\usepackage{tikz}
\usepackage{booktabs}
\usepackage{caption}
\usepackage{subcaption}
\usepackage{url}
%\usepackage{hyperref}

\usepackage{color}
\newcommand{\todo}[1]{\textcolor{red}{#1}}
\newcommand{\cs}[1]{\textcolor{green}{\textbf{CS}:#1}}


\usepackage{xspace}
\def\etal{\emph{et al.}\xspace}
\def\ie{\emph{i.e.}\xspace}
\def\eg{\emph{e.g.}\xspace}

\setlength{\pdfpagewidth}{8.5in}
\setlength{\pdfpageheight}{11in}

\newfont{\mycrnotice}{ptmr8t at 7pt}
\newfont{\myconfname}{ptmri8t at 7pt}
\let\crnotice\mycrnotice%
\let\confname\myconfname%

%\permission{Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from permissions@acm.org.}
%\conferenceinfo{ICMR'16}{June 06--09, 2016, New York, United States of America.}
%\copyrightetc{Copyright \copyright~2016 ACM \the\acmcopyr}
%\crdata{?...\$15.00.\\
%http://dx.doi.org/?/?}

\clubpenalty=10000
\widowpenalty = 10000

\begin{document}
%
% --- Author Metadata here ---
%\conferenceinfo{ICMR'15 }{June 23-26, 2015, Shanghai, China}
%\CopyrightYear{2015} % Allows default copyright year (20XX) to be over-ridden - IF NEED BE.
%\crdata{978-1-4503-3274-3/15/06}%\\
%http://dx.doi.org/10.1145/2671188.2749404
%}  % Allows default copyright data (0-89791-88-6/97/05) to be over-ridden - IF NEED BE.
% --- End of Author Metadata ---

%\title{Video Event Detection and Recounting\\by Fully Leveraging the ImageNet Hierarchy}
%\title{Video Event Detection\\by Fully Leveraging the ImageNet Hierarchy}
%\title{Video Event Detection\\using Reorganized Imagenet Hierarchies}
%
%\title{Pre-Training with ImageNet Reshuffles for Video Event Detection} 
%
%\title{Reshuffled ImageNet Pre-Training Improves Video Event Detection} 
%\title{Reorganized ImageNet Pre-Training Improves Video Event Detection}
\title{The ImageNet Shuffle:\\Reorganized Pre-training for Video Event Detection}

%
% You need the command \numberofauthors to handle the 'placement
% and alignment' of the authors beneath the title.
%
% For aesthetic reasons, we recommend 'three authors at a time'
% i.e. three 'name/affiliation blocks' be placed beneath the title.
%
% NOTE: You are NOT restricted in how many 'rows' of
% "name/affiliations" may appear. We just ask that you restrict
% the number of 'columns' to three.
%
% Because of the available 'opening page real-estate'
% we ask you to refrain from putting more than six authors
% (two rows with three columns) beneath the article title.
% More than six makes the first-page appear very cluttered indeed.
%
% Use the \alignauthor commands to handle the names
% and affiliations for an 'aesthetic maximum' of six authors.
% Add names, affiliations, addresses for
% the seventh etc. author(s) as the argument for the
% \additionalauthors command.
% These 'additional authors' will be output/set for you
% without further effort on your part as the last section in
% the body of your article BEFORE References or any Appendices.

% \numberofauthors{1} %  in this sample file, there are a *total*
% of EIGHT authors. SIX appear on the 'first-page' (for formatting
% reasons) and the remaining two appear in the \additionalauthors section.
%
% \author{
% You can go ahead and credit any number of authors here,
% e.g. one 'row of three' or two rows (consisting of one row of three
% and a second row of one, two or three).
%
% The command \alignauthor (no curly braces needed) should
% precede each author name, affiliation/snail-mail address and
% e-mail address. Additionally, tag each line of
% affiliation/address with \affaddr, and tag the
% e-mail address with \email.
%
% 1st. author
%\alignauthor

\numberofauthors{1}
\author{
Pascal Mettes, Dennis C. Koelma, Cees G.~M. Snoek\\\\
\affaddr{University of Amsterdam}
}
%\email{\{P.S.M.Mettes, J.C.vanGemert, S.H.Cappallo, Thomas.Mensink, cgmsnoek\}@uva.nl}
%}

%Ben Trovato\titlenote{Dr.~Trovato insisted his name be first.}\\
%       \affaddr{Institute for Clarity in Documentation}\\
%       \affaddr{1932 Wallamaloo Lane}\\
%       \affaddr{Wallamaloo, New Zealand}\\
%       \email{trovato@corporation.com}
%% 2nd. author
%\alignauthor
%G.K.M. Tobin\titlenote{The secretary disavows
%any knowledge of this author's actions.}\\
%       \affaddr{Institute for Clarity in Documentation}\\
%       \affaddr{P.O. Box 1212}\\
%       \affaddr{Dublin, Ohio 43017-6221}\\
%       \email{webmaster@marysville-ohio.com}
%% 3rd. author
%\alignauthor Lars Th{\o}rv{\"a}ld\titlenote{This author is the
%one who did all the really hard work.}\\
%       \affaddr{The Th{\o}rv{\"a}ld Group}\\
%       \affaddr{1 Th{\o}rv{\"a}ld Circle}\\
%       \affaddr{Hekla, Iceland}\\
%       \email{larst@affiliation.org}
%\and  % use '\and' if you need 'another row' of author names
%% 4th. author
%\alignauthor Lawrence P. Leipuner\\
%       \affaddr{Brookhaven Laboratories}\\
%       \affaddr{Brookhaven National Lab}\\
%       \affaddr{P.O. Box 5000}\\
%       \email{lleipuner@researchlabs.org}
%% 5th. author
%\alignauthor Sean Fogarty\\
%       \affaddr{NASA Ames Research Center}\\
%       \affaddr{Moffett Field}\\
%       \affaddr{California 94035}\\
%       \email{fogartys@amesres.org}
%% 6th. author
%\alignauthor Charles Palmer\\
%       \affaddr{Palmer Research Laboratories}\\
%       \affaddr{8600 Datapoint Drive}\\
%       \affaddr{San Antonio, Texas 78229}\\
%       \email{cpalmer@prl.com}
%}

\maketitle
\input{abstract}

% A category with the (minimum) three required fields
%\category{--}{Artificial Intelligence}{Activity recognition and understanding}
%\category{I.2.10}{Vision and Scene Understanding}{Video analysis}
%
%%\terms{--}
%
\keywords{Event Detection; Video Representation Learning}

\input{introduction}

\input{relatedwork}

\input{hierarchy}

\input{experiments}

\input{conclusions}

\section*{Acknowledgements}
This research is supported by the STW STORY project.
\vspace{-0.2cm}

\bibliographystyle{abbrv}
\bibliography{the-imagenet-shuffle-arxiv}

\end{document}

the-imagenet-shuffle-arxiv.bbl
\begin{thebibliography}{10}

\bibitem{chang2015searching}
X.~Chang, Y.-L. Yu, Y.~Yang, and A.~G. Hauptmann.
\newblock Searching persuasively: Joint event detection and evidence recounting
  with limited supervision.
\newblock In {\em MM}, 2015.

\bibitem{imagenet09}
J.~Deng, W.~Dong, R.~Socher, L.-J. Li, K.~Li, and L.~Fei-Fei.
\newblock {ImageNet}: A large-scale hierarchical image database.
\newblock In {\em CVPR}, 2009.

\bibitem{habibian14}
A.~Habibian, T.~Mensink, and C.~G.~M. Snoek.
\newblock Videostory: A new multimedia embedding for few-example recognition
  and translation of events.
\newblock In {\em MM}, 2014.

\bibitem{habibian13}
A.~Habibian, K.~E.~A. van~de Sande, and C.~G.~M. Snoek.
\newblock Recommendations for video event recognition using concept
  vocabularies.
\newblock In {\em ICMR}, 2013.

\bibitem{jegou2012aggregating}
H.~J{\'e}gou, F.~Perronnin, M.~Douze, J.~Sanchez, P.~Perez, and C.~Schmid.
\newblock Aggregating local image descriptors into compact codes.
\newblock {\em TPAMI}, 34(9):1704--1716, 2012.

\bibitem{jia2014caffe}
Y.~Jia, E.~Shelhamer, J.~Donahue, S.~Karayev, J.~Long, R.~Girshick,
  S.~Guadarrama, and T.~Darrell.
\newblock Caffe: Convolutional architecture for fast feature embedding.
\newblock In {\em MM}, 2014.

\bibitem{jiang2015fast}
L.~Jiang, S.-I. Yu, D.~Meng, Y.~Yang, T.~Mitamura, and A.~G. Hauptmann.
\newblock Fast and accurate content-based semantic search in 100m internet
  videos.
\newblock pages 49--58, 2015.

\bibitem{karpathy2014large}
A.~Karpathy, G.~Toderici, S.~Shetty, T.~Leung, R.~Sukthankar, and L.~Fei-Fei.
\newblock Large-scale video classification with convolutional neural networks.
\newblock In {\em CVPR}, 2014.

\bibitem{krizhevsky2012imagenet}
A.~Krizhevsky, I.~Sutskever, and G.~E. Hinton.
\newblock Imagenet classification with deep convolutional neural networks.
\newblock In {\em NIPS}, 2012.

\bibitem{lan2014multimedia}
Z.-Z. Lan, L.~Bao, S.-I. Yu, W.~Liu, and A.~G. Hauptmann.
\newblock Multimedia classification and event detection using double fusion.
\newblock {\em MTA}, 71(1):333--347, 2014.

\bibitem{mazloom2014conceptlets}
M.~Mazloom, E.~Gavves, and C.~G. Snoek.
\newblock Conceptlets: Selective semantics for classifying video events.
\newblock {\em TMM}, 16(8):2214--2228, 2014.

\bibitem{mettes2015bag}
P.~Mettes, J.~C. van Gemert, S.~Cappallo, T.~Mensink, and C.~G.~M. Snoek.
\newblock Bag-of-fragments: Selecting and encoding video fragments for event
  detection and recounting.
\newblock In {\em ICMR}, 2015.

\bibitem{miller1995wordnet}
G.~A. Miller.
\newblock Wordnet: a lexical database for english.
\newblock {\em Comm. ACM}, 38(11):39--41, 1995.

\bibitem{myers2014evaluating}
G.~K. Myers, R.~Nallapati, J.~van Hout, S.~Pancoast, R.~Nevatia, C.~Sun,
  A.~Habibian, D.~C. Koelma, K.~E.~A. van~de Sande, A.~W.~M. Smeulders, and
  C.~G.~M. Snoek.
\newblock Evaluating multimedia features and fusion for example-based event
  detection.
\newblock {\em MVA}, 25(1):17--32, 2014.

\bibitem{nagel2015event}
M.~Nagel, T.~Mensink, and C.~G.~M. Snoek.
\newblock Event fisher vectors: Robust encoding visual diversity of visual
  streams.
\newblock In {\em BMVC}, 2015.

\bibitem{natarajan2012multimodal}
P.~Natarajan, S.~Wu, S.~Vitaladevuni, X.~Zhuang, S.~Tsakalidis, U.~Park,
  R.~Prasad, and P.~Natarajan.
\newblock Multimodal feature fusion for robust event detection in web videos.
\newblock In {\em CVPR}, 2012.

\bibitem{over14}
P.~Over et~al.
\newblock Trecvid 2014 -- an overview of the goals, tasks, data, evaluation
  mechanisms and metrics.
\newblock In {\em TRECVID Workshop}, 2014.

\bibitem{russakovsky2014imagenet}
O.~Russakovsky, J.~Deng, H.~Su, J.~Krause, S.~Satheesh, S.~Ma, Z.~Huang,
  A.~Karpathy, A.~Khosla, M.~Bernstein, et~al.
\newblock Imagenet large scale visual recognition challenge.
\newblock {\em IJCV}, 115(3):211--252, 2015.

\bibitem{sanchez2013image}
J.~S{\'a}nchez, F.~Perronnin, T.~Mensink, and J.~Verbeek.
\newblock Image classification with the fisher vector: Theory and practice.
\newblock {\em IJCV}, 105(3):222--245, 2013.

\bibitem{simonyan2014very}
K.~Simonyan and A.~Zisserman.
\newblock Very deep convolutional networks for large-scale image recognition.
\newblock {\em ICLR}, 2015.

\bibitem{sun142}
C.~Sun, B.~Burns, R.~Nevatia, C.~G.~M. Snoek, B.~Bolles, G.~Myers, W.~Wang, and
  E.~Yeh.
\newblock Isomer: Informative segment observations for multimedia event
  recounting.
\newblock In {\em ICMR}, 2014.

\bibitem{szegedy2014going}
C.~Szegedy, W.~Liu, Y.~Jia, P.~Sermanet, S.~Reed, D.~Anguelov, D.~Erhan,
  V.~Vanhoucke, and A.~Rabinovich.
\newblock Going deeper with convolutions.
\newblock {\em CVPR}, 2015.

\bibitem{vreeswijk2012all}
D.~T.~J. Vreeswijk, C.~G.~M. Snoek, K.~E.~A. van~de Sande, and A.~W.~M.
  Smeulders.
\newblock All vehicles are cars: subclass preferences in container concepts.
\newblock In {\em ICMR}, 2012.

\bibitem{wang13}
H.~Wang and C.~Schmid.
\newblock Action recognition with improved trajectories.
\newblock In {\em ICCV}, 2013.

\bibitem{xu2014discriminative}
Z.~Xu, Y.~Yang, and A.~G. Hauptmann.
\newblock A discriminative cnn video representation for event detection.
\newblock {\em CVPR}, 2015.

\bibitem{ye2015eventnet}
G.~Ye, Y.~Li, H.~Xu, D.~Liu, and S.-F. Chang.
\newblock Eventnet: A large scale structured concept library for complex event
  detection in video.
\newblock In {\em MM}, 2015.

\end{thebibliography}
