---
abstract: 'First-person vision is gaining interest as it offers a unique viewpoint on people’s interaction with objects, their attention, and even intention. However, progress in this challenging domain has been relatively slow due to the lack of sufficiently large datasets. In this paper, we introduce , a large-scale egocentric video benchmark recorded by 32 participants in their native kitchen environments. Our videos depict **non-scripted** daily activities: we simply asked each participant to start recording every time they entered their kitchen. Recording took place in 4 cities (in North America and Europe) by participants belonging to 10 different nationalities, resulting in highly diverse cooking styles. Our dataset features 55 hours of video consisting of 11.5M frames, which we densely labelled for a total of $39.6$K action segments and $454.3$K object bounding boxes. Our annotation is unique in that we had the participants narrate their own videos (after recording), thus reflecting true intention, and we crowd-sourced ground-truths based on these. We describe our object, action and anticipation challenges, and evaluate several baselines over two test splits, *seen* and *unseen* kitchens.'
author:
- Dima Damen$^1$
- Hazel Doughty$^1$
- Giovanni Maria Farinella$^2$
- Sanja Fidler$^3$
- Antonino Furnari$^2$
- Evangelos Kazakos$^1$
- Davide Moltisanti$^1$
- |
    \
    Jonathan Munro$^1$
- Toby Perrett$^1$
- Will Price$^1$
- Michael Wray$^1$
title: |
    Scaling Egocentric Vision:\
    The Dataset
---

![From top: Frames from the 32 environments; Narrations by participants used to annotate action segments; Active object bounding box annotations[]{data-label="fig:epic-wall"}](Figures-TP/wall.jpg "fig:"){width="1\columnwidth"}\
![From top: Frames from the 32 environments; Narrations by participants used to annotate action segments; Active object bounding box annotations[]{data-label="fig:epic-wall"}](Figures-DM/annotations-sample.pdf "fig:"){width="1\columnwidth"}

Introduction {#sec:introduction}
============

In recent years, we have seen significant progress in many domains such as image classification [@resnet], object detection [@ren2015faster], captioning [@Karpathy14] and visual question-answering [@VQA]. This success has in large part been due to advances in deep learning [@alexnet] as well as the availability of large-scale image benchmarks [@pascal; @imagenet; @coco; @ade]. While gaining attention, work in video understanding has been more scarce, mainly due to the lack of annotated datasets. This has been changing recently, with the release of the action classification benchmarks such as [@Goyal2017; @Haija2016; @zhao2017slac; @moviedescription; @movieqa; @fouhey2017lifestyle]. With the exception of [@movieqa], most of these datasets contain videos that are very short in duration, i.e., only a few seconds long, focusing on a single action. Charades [@Sigurdsson2016] makes a step towards activity recognition by collecting 10K videos of humans performing various tasks in their home. While this dataset is a nice attempt to collect daily actions, the videos have been recorded in a scripted way, by asking AMT workers to act out a script in front of the camera. This makes the videos look oftentimes less natural, and they also lack the progression and multi-tasking of actions that occur in real life.

Here we focus on first-person vision, which offers a unique viewpoint on people’s daily activities. This data is rich as it reflects our goals and motivation, ability to multi-task, and the many different ways to perform a variety of important, but mundane, everyday tasks (such as cleaning the dishes). Egocentric data has also recently been proven valuable for human-to-robot imitation learning [@Nair2017; @Zhang2018], and has a direct impact on HCI applications. However, datasets to evaluate first-person vision algorithms [@EGTEA; @sigurdsson2018charadesego; @Damen2014a; @Fathi2012; @Pirsiavash2012; @de2008guide] have been significantly smaller in size than their third-person counterparts, often captured in a single environment [@EGTEA; @Damen2014a; @Fathi2012; @de2008guide]. Daily interactions from wearable cameras are also scarcely available online, making this a largely unavailable source of information.

In this paper, we introduce , a large-scale egocentric dataset. Our data was collected by 32 participants, belonging to 10 nationalities, in their native kitchens (Fig. \[fig:epic-wall\]). The participants were asked to capture all their daily kitchen activities, and record sequences regardless of their duration. The recordings, which include both video and sound, not only feature the typical interactions with one’s own kitchenware and appliances, but importantly show the natural multi-tasking that one performs, like washing a few dishes amidst cooking. Such parallel-goal interactions have not been captured in existing datasets, making this both a more realistic as well as a more challenging set of recordings. A video introduction to the recordings is available at: .

Altogether, has 55hrs of recording, densely annotated with start/end times for each action/interaction, as well as bounding boxes around objects subject to interaction. We describe our object, action and anticipation challenges, and report baselines in two scenarios, i.e., *seen* and *unseen* kitchens. The dataset and leaderboards to track the community’s progress on all challenges, with held out test ground-truth are at: .

Related Datasets
================

We compare to four commonly-used [@Damen2014a; @Fathi2012; @Pirsiavash2012; @de2008guide] and two recent [@EGTEA; @sigurdsson2018charadesego] egocentric datasets in Table \[tab:datasets\], as well as six third-person activity-recognition datasets [@fouhey2017lifestyle; @Sigurdsson2016; @Zhou2017; @Kuehne2014; @Stein2013; @Rohrbach2012] that focus on object-interaction activities. We exclude egocentric datasets that focus on inter-person interactions [@Alletto2015; @Fathi2012b; @Ryoo2013], as these target a different research question.

A few datasets aim at capturing activities in native environments, most of which are recorded in third-person [@Goyal2017; @fouhey2017lifestyle; @Sigurdsson2016; @sigurdsson2018charadesego; @Kuehne2014].  [@Kuehne2014] focuses on cooking dishes based on a list of breakfast recipes. In [@fouhey2017lifestyle], short segments linked to interactions with 30 daily objects are collected by querying YouTube, while [@Goyal2017; @Sigurdsson2016; @sigurdsson2018charadesego] are scripted – subjects are requested to enact a crowd-sourced storyline [@Sigurdsson2016; @sigurdsson2018charadesego][^1] or a given action [@Goyal2017], which oftentimes results in less natural looking actions. All egocentric datasets similarly use scripted activities, i.e. people are told what actions to perform. When following instructions, participants perform steps in a sequential order, as opposed to the more natural real-life scenarios addressed in our work, which involve multi-tasking, searching for an item, thinking what to do next, changing one’s mind or even unexpected surprises. is most closely related to the ADL dataset [@Pirsiavash2012] which also provides egocentric recordings in native environments. However, our dataset is substantially larger: it has 11.5M frames vs 1M in ADL, 90x more annotated action segments, and 4x more object bounding boxes, making it the largest first-person dataset to date.

The Dataset {#sec:dataset}
===========

In this section, we describe our data collection and annotation pipeline. We also present various statistics, showcasing different aspects of our collected data.

Data Collection {#sec:recording}
---------------

The dataset was recorded by 32 individuals in 4 cities in different countries (in North America and Europe): 15 in Bristol/UK, 8 in Toronto/Canada, 8 in Catania/Italy and 1 in Seattle/USA between May and Nov 2017. Participants were asked to capture all kitchen visits *for three consecutive days*, with the recording starting immediately before entering the kitchen, and only stopped before leaving the kitchen. They recorded the dataset voluntarily and were not financially rewarded. The participants were asked to be in the kitchen alone for all the recordings, thus capturing only one-person activities. We also asked them to remove all items that would disclose their identity such as portraits or mirrors. Data was captured using a head-mounted GoPro with an adjustable mounting to control the viewpoint for different environments and participants’ heights. Before each recording, the participants checked the battery life and viewpoint, using the GoPro Capture app, so that their stretched hands were approximately located at the middle of the camera frame. The camera was set to linear field of view, 59.94[*fps*]{} and Full HD resolution of 1920x1080, however some subjects made minor changes like wide or ultra-wide FOV or resolution, as they recorded multiple sequences in their homes, and thus were switching the device off and on over several days. Specifically, 1% of the videos were recorded at 1280x720 and 0.5% at 1920x1440. Also, 1% at 30[*fps*]{}, 1% at 48[*fps*]{} and 0.2% at 90[*fps*]{}.

The recording lengths varied depending on the participant’s kitchen engagement. On average, people recorded for 1.7hrs, with the maximum being 4.6hrs. Cooking a single meal can span multiple sequences, depending on whether one stays in the kitchen, or leaves and returns later. On average, each participant recorded 13.6 sequences. Figure \[fig:statsCollect\] presents statistics on time of day using the local time of the recording, high-level goals and sequence durations.

Since crowd-sourcing annotations for such long videos is very challenging, we had our original participants do a coarse first annotation. Each participant was asked to watch their videos, after completing all recordings, and narrate the actions carried out, using a hand-held recording device. We opted for a sound recording rather than written captions as this is arguably much faster for the participants, who were thus more willing to provide these annotations. These are analogous to a *live commentary* of the video. The general instructions for narrations are listed in Fig. \[fig:narrationInstruction\]. The participant narrated in English if sufficiently fluent or in their native language. In total, 5 languages were used: 17 narrated in English, 7 in Italian, 6 in Spanish, 1 in Greek and 1 in Chinese. Figure \[fig:statsCollect\] shows wordles of the most frequent words in each language.

![[**Top**]{} (left to right): time of day of the recording, pie chart of high-level goals, histogram of sequence durations and dataset logo; [**Bottom**]{}: Wordles of narrations in native languages (English, Italian, Spanish, Greek and Chinese)[]{data-label="fig:statsCollect"}](Figures-WP/epic-hours.pdf "fig:"){width="0.25\columnwidth"} ![[**Top**]{} (left to right): time of day of the recording, pie chart of high-level goals, histogram of sequence durations and dataset logo; [**Bottom**]{}: Wordles of narrations in native languages (English, Italian, Spanish, Greek and Chinese)[]{data-label="fig:statsCollect"}](Figures-DM/videoGoals.pdf "fig:"){width="0.28\columnwidth"} ![[**Top**]{} (left to right): time of day of the recording, pie chart of high-level goals, histogram of sequence durations and dataset logo; [**Bottom**]{}: Wordles of narrations in native languages (English, Italian, Spanish, Greek and Chinese)[]{data-label="fig:statsCollect"}](Figures-D/SeqLengths_All_Logo.png "fig:"){width="0.45\columnwidth"}\
[![[**Top**]{} (left to right): time of day of the recording, pie chart of high-level goals, histogram of sequence durations and dataset logo; [**Bottom**]{}: Wordles of narrations in native languages (English, Italian, Spanish, Greek and Chinese)[]{data-label="fig:statsCollect"}](Figures-TP/english.pdf "fig:"){width="0.2\columnwidth"}]{} [![[**Top**]{} (left to right): time of day of the recording, pie chart of high-level goals, histogram of sequence durations and dataset logo; [**Bottom**]{}: Wordles of narrations in native languages (English, Italian, Spanish, Greek and Chinese)[]{data-label="fig:statsCollect"}](Figures-TP/n_italian.pdf "fig:"){width="0.2\columnwidth"}]{} [![[**Top**]{} (left to right): time of day of the recording, pie chart of high-level goals, histogram of sequence durations and dataset logo; [**Bottom**]{}: Wordles of narrations in native languages (English, Italian, Spanish, Greek and Chinese)[]{data-label="fig:statsCollect"}](Figures-TP/n_spanish.pdf "fig:"){width="0.2\columnwidth"}]{} [![[**Top**]{} (left to right): time of day of the recording, pie chart of high-level goals, histogram of sequence durations and dataset logo; [**Bottom**]{}: Wordles of narrations in native languages (English, Italian, Spanish, Greek and Chinese)[]{data-label="fig:statsCollect"}](Figures-TP/n_greek.pdf "fig:"){width="0.2\columnwidth"}]{} [![[**Top**]{} (left to right): time of day of the recording, pie chart of high-level goals, histogram of sequence durations and dataset logo; [**Bottom**]{}: Wordles of narrations in native languages (English, Italian, Spanish, Greek and Chinese)[]{data-label="fig:statsCollect"}](Figures-TP/n_chinese.pdf "fig:"){width="0.2\columnwidth"}]{}

\[fig:transcriptfiles\]

Our decision to collect narrations from the participants themselves is because they are the most qualified to label the activity compared to an independent observer, as they were the ones performing the actions. We opted for a post-recording narration such that the participant performs her/his daily activities undisturbed, without being concerned about labelling.

We tested several automatic audio-to-text APIs [@googleSpeechApi; @ibmWatson; @cmuSphinx], which failed to produce accurate transcriptions as these expect a relevant corpus and complete sentences for context. We thus collected manual transcriptions via Amazon Mechanical Turk (AMT), and used the YouTube’s automatic closed caption alignment tool to produce accurate timings. For non-English narrations, we also asked AMT workers to translate the sentences. To make the job more suitable for AMT, narration audio files are split by removing silence below a pre-specified decibel threshold (after compression and normalisation). Speech chunks are then combined into HITs with a duration of around 30 seconds each. To ensure consistency, we submit the same HIT three times and select the ones with an edit distance of 0 to at least one other HIT. We manually corrected cases when there was no agreement. Examples of transcribed and timed narrations are provided in Table \[fig:transcriptfiles\]. The participants were also asked to provide one sentence per sequence describing the overall goal or activity that took place.

In total, we collected $39,596$ action narrations, corresponding to a narration every $4.9s$ in the video. The average number of words per phrase is $2.8$ words. These narrations give us an initial labelling of all actions with rough temporal alignment, obtained from the timestamp of the audio narration with respect to the video. However, narrations are also not a perfect source of ground-truth:

-   The narrations can be incomplete, i.e., the participants were selective in which actions they chose to narrate. We noticed that they labelled the ‘open’ actions more than their counter-action ‘close’, as the narrator’s attention has already moved to the next goal. We consider this phenomena in our evaluation, by only evaluating actions that have been narrated.

-   Temporally, the narrations are belated, after the action takes place. This is adjusted using ground-truth action segments (see Sec. \[subsec:strongAction\]).

-   Participants use their own vocabulary and free language. While this is a challenging issue, we believe it is important to push the community to go beyond the pre-selected list of labels (also argued in [@ade]). We here resolve this issue by grouping verbs and nouns into minimally overlapping classes (see Sec. \[subsec:clusters\]).

Action Segment Annotations {#subsec:strongAction}
--------------------------

For each narrated sentence, we adjust the start and end times of the action using AMT. To ensure the annotators are trained to perform temporal localisation, we use a clip from our previous work’s understanding [@Moltisanti2017] that explains temporal bounds of actions. Each HIT is composed of a maximum of 10 consecutive narrated phrases $p_i$, where annotators label $A_{i} = [t_{s_i}, t_{e_i}]$ as the start and end times of the $i^{th}$ action. Two constraints were added to decrease the amount of noisy annotations: (1) action has to be at least 0.5 seconds in length; (2) action cannot start before the preceding action’s start time. Note that consecutive actions are allowed to overlap. Moreover, the annotators could indicate that the action does not appear in the video. This handles occluded, impossible to distinguish or out-of-bounds cases.

To ensure consistency, we ask $\mathcal{K}_a=4$ annotators to annotate each HIT. Given one annotation $A_i(j)$ ($i$ is the action and $j$ indexes the annotator), we calculate the agreement as follows: ${\alpha_i(j) = \frac{1}{K_a} \sum_{k=1}^{\mathcal{K}_a} \text{IoU} (A_i(j), A_i(k))}$. We first find the annotator with the maximum agreement $\hat{j} = \arg\max_j \alpha_i(j)$, and find $\hat{k} = \arg\max_k \text{IoU}(A_i(\hat{j}), A_i(k))$. The ground-truth action segment $A_i$ is then defined as: $$\small{
A_i =\begin{cases}
\text{Union}(A_i(\hat{j}), A_i(\hat{k})), & \text{if  IoU} (A_i(\hat{j}), A_i(\hat{k}))>0.5\\
A_i(\hat{j}), & \text{otherwise}
\end{cases}}
\label{eq:actionunion}$$ We thus combine two annotations when they have a strong agreement, since in some cases the single (best) annotation results in a too tight of a segment. Figure \[fig:annotations\_example\] shows examples of combining annotations.

![Object annotation from three AMT workers (orange, blue and green). The green participant’s annotations are selected as the final annotations[]{data-label="fig:object_ann"}](Figures-D/annotations_video_only2.png){width="\linewidth"}

![Object annotation from three AMT workers (orange, blue and green). The green participant’s annotations are selected as the final annotations[]{data-label="fig:object_ann"}](Figures-HD/annotation_consensus){width="\linewidth"}

In total, we collected such labels for $39,564$ action segments (lengths: ${\mu=3.7s}$, ${\sigma=5.6s}$). These represent 99.9% of narrated segments. The missed annotations were those labelled as “not visible” by the annotators, though mentioned in narrations.

Active Object Bounding Box Annotations {#subsec:strongObject}
--------------------------------------

The narrated *nouns* correspond to objects relevant to the action [@Lee2012; @Damen2014a]. Assume $\mathcal{O}_i$ is the set of one or more nouns in the phrase $p_i$ associated with the action segment $A_i = [t_{s_i}, t_{e_i}]$. We consider each frame $f$ within $[t_{s_i}-2s,t_{e_i}+2s]$ as a potential frame to annotate the bounding box(es), for each object in $\mathcal{O}_i$. We build on the interface from [@tangseng2017] for annotating bounding boxes on AMT. Each HIT aims to get an annotation for one object, for the maximum duration of $25s$, which corresponds to $50$ consecutive frames at $2$[*fps*]{}. The annotator can also note that the object does not exist in $f$. We particularly ask the same annotator to annotate consecutive frames to avoid subjective decisions on the extents of objects. We also assess annotators’ quality by ensuring that the annotators obtain an ${\textrm{IoU} \ge 0.7}$ on two golden annotations at the start of every HIT. We request $\mathcal{K}_o = 3$ workers per HIT, and select the one with maximum agreement $\beta$: $$\beta(q) = \sum_f \max\limits_{j \ne q}^{\mathcal{K}_o}\, \max_{k,l}\, \text{IoU}(\text{BB}(j, f, k), \text{BB}(q, f, l))$$

where $\text{BB}(q,f, k)$ is the $k^{th}$ bounding box annotation by annotator $q$ in frame $f$. Ties are broken by selecting the worker who provides the tighter bounding boxes. Figure \[fig:object\_ann\] shows multiple annotations for four keyframes in a sequence.

Overall, 77% of requested annotations resulted in at least one bounding box. In total, we collected 454,255 bounding boxes ($\mu = 1.64$ boxes/frame, ${\sigma = 0.92}$). Sample action segments and object bounding boxes are shown in Fig. \[fig:timeline\_example\].

![Sample consecutive action segments with keyframe object annotations[]{data-label="fig:timeline_example"}](Figures-DM/timeline_dm.png){width="\linewidth"}

Verb and Noun Classes {#subsec:clusters}
---------------------

Since our participants annotated using free text in multiple languages, a variety of verbs and nouns have been collected. We group these into classes with minimal semantic overlap, to accommodate the more typical approaches to multi-class detection and recognition where each example is believed to belong to one class only. We estimate Part-of-Speech (POS), using SpaCy’s English core web model. We select the first verb in the sentence, and find all nouns in the sentence excluding any that match the chosen verb. When a noun is absent or replaced by a pronoun (*e.g. ‘it’*), we use the noun from the directly preceding narration (e.g. $p_i$: ‘rinse cup’, $p_{i+1}$: ‘place it to dry’).

![image](Figures-D/verb_stats2.png){width="100.00000%"}\
![image](Figures-D/nouns-stats_final.png){width="100.00000%"}\
![image](Figures-D/nouns-stats2.png){width="100.00000%"}\
![image](Figures-D/bb-resolution.png){width="100.00000%"}

We refer to the set of minimally-overlapping verb classes as $C_V$, and similarly $C_N$ for nouns. We attempted to automate the clustering of verbs and nouns using combinations of WordNet [@miller1995wordnet], Word2Vec [@mikolov2013efficient], and Lesk algorithm [@Banerjee2002], however, due to limited context there were too many meaningless clusters. We thus elected to manually cluster the verbs and semi-automatically cluster the nouns. We preprocessed the compound nouns *e.g.* ‘pizza cutter’ as a subset of the second noun *e.g.* ‘cutter’. We then manually adjusted the clustering, merging the variety of names used for the same object, *e.g.* ‘cup’ and ‘mug’, as well as splitting some base nouns, *e.g.* ‘washing machine’ vs ‘coffee machine’.

In total, we have 125 $C_V$ classes and 331 $C_N$ classes. Table \[tab:classes\] shows a sample of grouped verbs and nouns into classes. These classes are used in all three defined challenges. In Fig. \[fig:stats\], we show $C_V$ ordered by frequency of occurrence in action segments, as well as $C_N$ ordered by number of annotated bounding boxes. These are grouped into 19 super categories, of which 9 are food and drinks, with the rest containing kitchen essentials from appliances to cutlery. Co-occurring classes are presented in Fig. \[fig:co-occur\].

Annotation Quality Assurance
----------------------------

To analyse the quality of annotations, we choose 300 random samples, and manually assess correctness. We report:

-   **Action Segment Boundaries ($A_i$)**: We check that the start/end times fully enclose the action boundaries, with any additional frames not part of other actions - error: *5.7%*.

-   **Object Bounding Boxes ($\mathcal{O}_i$)**: We check that the bounding box encapsulates the object or its parts, with minimal overlap with other objects, and that all instances of the class in the frame have been labelled – error: *6.3%*.

-   **Verb classes ($C_V$):** We check that the verb class is correct – error: *3.3%*.

-   **Noun classes ($C_N$):** We check that the noun class is correct – error : *6.0%*.

These error rates are comparable to recently published datasets [@zhao2017slac].

Benchmarks and Baseline Results {#sec:baselines}
===============================

offers a variety of potential challenges from routine understanding, to activity recognition and object detection. As a start, we define three challenges for which we provide baseline results, and avail online leaderboards. For the evaluation protocols, we hold out ground truth annotations for 27% of the data (Table \[tab:s1\_stats\]). We particularly aim to assess the generalizability to novel environments, and we thus structured our test set to have a collection of *seen* and previously *unseen* kitchens:

**Seen Kitchens (S1):** In this split, each kitchen is seen in both training and testing, where roughly 80% of sequences are in training and 20% in testing. We do not split sequences, thus each sequence is in either training or testing.

**Unseen Kitchens (S2):** This divides the participants/kitchens so all sequences of the same kitchen are either in training or testing. We hold out the complete sequences for 4 participants for this testing protocol. The test set of S2 is only 7% of the dataset in terms of frame count, but the challenges remain considerable.

\[tab:classes\]

We now evaluate several existing methods on our benchmarks, to gain an understanding of how challenging our dataset is.

Object Detection Benchmark {#subsec:obj_challenges}
--------------------------

**Challenge:** This challenge focuses on object detection for all of our $C_N$ classes. Note that our annotations only capture the ‘active’ objects pre-, during- and post- interaction. We thus restrict the images evaluated per class to those where the object has been annotated. We particularly aim to break the performance down into multi-shot and few-shot class groups, so as to analyse the capabilities of the approaches to quickly learn novel objects (with only a few examples). Our challenge leaderboard reflects the methods’ abilities on both sets of classes.

**Method:** We evaluate object detection using Faster R-CNN [@ren2015faster] due to its state-of-the-art performance. Faster R-CNN uses a region proposal network (RPN) to first generate class agnostic object proposals, and then classifies these and outputs refined bounding box predictions. We use the implementation from [@objapi; @huang2017speed] with a base architecture of ResNet-101 [@resnet] pre-trained on MS-COCO [@coco].

**Implementation Details:** Learning rate is initialised to 0.0003 decaying by a factor of 10 after 90K and stopped after 120K iterations. We use a mini-batch size of 4 on 8 Nvidia P100 GPUs on a single compute node (Nvidia DGX-1) with distributed training and parameter synchronisation – i.e. overall mini-batch size of 32. As in [@ren2015faster], images are rescaled such that their shortest side is 600 pixels and the aspect ratio is maintained. We use a stride of 16 on the last convolution layer for feature extraction and for anchors we use 4 scales of 0.25, 0.5, 1.0 and 2.0; and aspect ratios of 1:1, 1:2 and 2:1. To reduce redundancy, NMS is used with an IoU threshold of 0.7. In training and testing we use 300 RPN proposals.

**Evaluation Metrics:** For each class, we only report results on $I^{c_n \in C_N}$, these are all images where class $c_n$ has been annotated. We use the mean average precision (mAP) metric from PASCAL VOC [@pascal], using IoU thresholds of 0.05, 0.5 and 0.75 similar to [@coco].

**Results:** We report results in Table \[tab:results\_obj\] for many-shot classes (those with ${\geq 100}$ bounding boxes in training) and few shot classes (with ${\geq 10}$ and ${< 100}$ bounding boxes in training), alongside AP for the 15 most frequent classes. There are a total of 202 many-shot classes and 88 few-shot classes. One can see that our objects are generally harder to detect than in most existing datasets, with performance at the standard IoU $>0.5$ below $40\%$. Even at a very small IoU threshold, the performance is relatively low. The more challenging classes are “meat”, “knife”, and “spoon”, despite being some of the most frequent ones. Notice that the performance for the low-shot regime is substantially lower than in the many-shot regime. This points to interesting challenges for the future. However, performances for the *Seen* and *Unseen* splits in object detection are comparable, thus showing generalization capability across environments.

Figure \[fig:qual\_obj\] shows qualitative results with detections shown in colour and ground truth shown in black. The examples in the right-hand column are failure cases.

\[tab:results\_obj\]

![Qualitative results for the object detection challenge[]{data-label="fig:qual_obj"}](Figures-HD/object_qualitative){width="1\linewidth" height="5cm"}

Action Recognition Benchmark {#sec:action_recognition_benchmark}
----------------------------

**Challenge:** Given an action segment $A_i = [t_{s_i}, t_{e_i}]$, we aim to classify the segment into its action class, where classes are defined as ${C_a = \{(c_v \in C_V, c_n \in C_N)\}}$, and $c_n$ is the first noun in the narration when multiple nouns are present. Note that our dataset supports more complex action-level challenges, such as action localisation in the videos of full duration. We decided to focus on the classification challenge first (the segment is provided) since most existing works tackle this challenge.

**Network Architecture:** We train the Temporal Segment Network (TSN) [@wang2016tsn] as a state-of-the-art architecture in action recognition, but adjust the output layer to predict both verb and noun classes jointly, with independent losses, as in [@kalogeiton2017]. We use the PyTorch implementation [@tsnpytorch] with the Inception architecture [@szegedy2015going], batch normalization [@ioffe2015batch] and pre-trained on ImageNet [@imagenet].

**Implementation Details:** We train both spatial and temporal streams, the latter on dense optical flow at 30[*fps*]{} extracted using the $\operatorname*{TV-L_1\!}$ algorithm [@zach2007duality] between RGB frames using the formulation $\operatorname*{TV-L_1\!}\left(I_{2t}, I_{2t+3}\right)$ to eliminate optical flicker, and released the computed flow as part of the dataset. We do not perform stratification or weighted sampling, allowing the dataset class imbalance to propagate into the mini-batch. We train each model on 8 Nvidia P100 GPUs on a single compute node (Nvidia DGX-1) for 80 epochs with a mini-batch size of 512. We set learning rate to 0.01 for spatial and 0.001 for temporal streams decreasing it by a factor of 10 after epochs 20 and 40. After averaging the 25 samples within the action segment each with 10 spatial croppings as in [@wang2016tsn], we fuse both streams by averaging class predictions with equal weights. All unspecified parameters use the same values as [@wang2016tsn].

**Evaluation Metrics:** We report two sets of metrics: aggregate and per-class, which are equivalent to the class-agnostic and class-aware metrics in [@zhao2017slac]. For aggregate metrics, we compute top-1 and top-5 accuracy for correct predictions of $c_v$, $c_n$ and their combination $(c_v, c_n)$ – we refer to these as ‘verb’, ‘noun’ and ‘action’. Accuracy is reported on the full test set. For per-class metrics, we compute precision and recall, for classes with more than 100 samples in training, then average the metrics across classes - these are 26 verb classes, 71 noun classes, and 819 action classes. Per-class metrics for smaller classes are $\approx 0$ as TSN is better suited for classes with sufficient training data.

**Results:** We report results in Table \[tab:results\_recognition\] for aggregate metrics and per-class metrics. We compare TSN (3 segments) to 2SCNN [@simonyan2014two] (1 segment), chance and largest class baselines. Fused results perform best or are comparable to the best stream (spatial/temporal). The challenge of getting both verb and noun labels correct remains significant for both *seen* (top-1 accuracy 20.5%) and *unseen* (top-1 accuracy 10.9%) environments. This implies that for many examples, we only get one of the two labels (verb/noun) right. Results also show that generalising to *unseen* environments is a harder challenge for actions than it is for objects. We give a breakdown per-class metrics for the 15 largest verb classes in Table \[tab:results\_recognition\_top\_verbs\].

![Qualitative results for the action recognition and anticipation challenges[]{data-label="fig:recognitionPrediction_qualitative"}](Figures-DM/actionRecognitionAnticipation.pdf){width="\linewidth"}

 \[fig:recognitionPrediction\_qualitative\] reports qualitative results, with success highlighted in green, and failures in red. In the first column both the verb and the noun are correctly predicted, in the second column one of them is correctly predicted, while in the third column both are incorrect. Challenging cases like distinguishing ‘adjust heat’ from turning it on, or pouring soy sauce vs oil are shown.

Action Anticipation Benchmark
-----------------------------

**Challenge:** Anticipating the next action is a well-mastered skill by humans, and automating it has direct implications in assertive living. Given any of the upcoming wearable system (e.g. Microsoft Hololens or Google Glass), anticipating the wearer’s next action, from a first-person view, could trigger smart home appliances, providing a seamless achievement of the wearer’s goals. Previous works have investigated different anticipation tasks from an egocentric perspective, e.g. predicting future localisation [@park2016egocentric] or next-active object [@furnari2017next]. We here consider the task of forecasting an action before it happens. Let $\tau_a$ be the ‘anticipation time’, how far in advance to recognise the action, and $\tau_o$ be the ‘observation time’, the length of the observed video segment preceding the action. Given an action segment $A_i=[t_{s_i},t_{e_i}]$, we predict the action class $C_a$ by observing the video segment *preceding* the action start time $t_{s_i}$ by $\tau_a$, that is $[t_{s_i}-(\tau_a+\tau_o),t_{s_i}-\tau_a]$.

**Network Architecture:** As in Sec. \[sec:action\_recognition\_benchmark\], we train TSN [@wang2016tsn] to provide baseline action anticipation results and compare with 2SCNN [@simonyan2014two]. We feed the model with the video segments preceding annotated actions and train it to predict verb and noun classes jointly as in [@kalogeiton2017]. Similarly to [@vondrick2016anticipating], we set $\tau_a=1s$. We report results with $\tau_o=1s$, and note that performance drops with longer segments.

**Implementation Details:** Models for both spatial and temporal modalities are trained using a single Nvidia Titan X with a batch size of 64, for $80$ epochs, setting the initial learning rate to $0.001$ and dropping it by a factor of 10 after $30$ and $60$ epochs. Fusion weights spatial and temporal streams with 0.6 and 0.4 respectively. All other parameters use the values specified in [@wang2016tsn].

**Evaluation Metrics:** We use the same evaluation metrics as in Sec. \[sec:action\_recognition\_benchmark\].

**Results:**  \[tab:results\_anticipation\] reports baseline results for the action anticipation challenge. As expected, this is a harder challenge than action recognition, and thus we note a drop in performance throughout. Unlike the case of action recognition, the flow stream and fusion do not generally improve performances. TSN often offers small, but consistent improvements over 2SCNN.

 \[fig:recognitionPrediction\_qualitative\] reports qualitative results. Success examples are highlighted in green, and failure cases in red. As the qualitative figure shows, the method over-predicts ‘put’ as the next action. Once an object is picked up, the learned model has a tendency to believe it will be put down next. Methods that focus on long-term understanding of the goal, as well as multi-scale history would be needed to circumvent such a tendency.

### Discussion:

The three defined challenges form the base for higher-level understanding of the wearer’s goals. We have shown that existing methods are still far from tackling these tasks with high precision, pointing to exciting future directions. Our dataset lends itself naturally to a variety of less explored tasks. We are planning to provide a wider set of challenges, including action localisation [@Yeung2018], video parsing [@Sigurdsson2016], visual dialogue [@visdial], goal completion [@Heidarivincheh2018] and skill determination [@Hazel2018] (e.g. how good are you at making your eggs for breakfast?). Since real-time performance is crucial in this domain, our leaderboard will reflect this, pressing the community to come up with efficient and effective solutions.

Conclusion and Future Work
==========================

We present the largest and most varied dataset in egocentric vision to date, , captured in participants’ native environments. We collect 55 hours of video data recorded on a head-mounted GoPro, and annotate it with narrations, action segments and object annotations using a pipeline that starts with live commentary of recorded videos by the participants themselves. Baseline results on object detection, action recognition and anticipation challenges show the great potential of the dataset for pushing approaches that target fine-grained video understanding to new frontiers.

Dataset Release: {#dataset-release .unnumbered}
================

-   Dataset sequences, extracted frames and optical flow are available at:\

-   Annotations, challenge leader-board results and updates and news are available at:

Acknowledgment {#acknowledgment .unnumbered}
==============

The authors would like to thank all 32 subjects who participated in the dataset collection.

The dataset annotation and release has been sponsored by a charitable donation from Nokia Technologies and the University of Bristol’s Jean Golding Institute.

Research at the University of Bristol is supported by EPSRC DTP, EPSRC GLANCE (EP/N013964/1) and EPSRC LOCATE (EP/N033779/1).

Research at the University of Catania is sponsored by Piano della Ricerca 2016-2018 – linea di Intervento 2 of DMI.

The object detection benchmark baseline results have been helped by code from, and discussions with, Davide Acu[ñ]{}a.

[10]{} \[1\][`#1`]{} \[1\][https://doi.org/\#1]{}

Abu[-]{}El[-]{}Haija, S., Kothari, N., Lee, J., Natsev, P., Toderici, G., Varadarajan, B., Vijayanarasimhan, S.: [YouTube-8M: [A]{} Large-Scale Video Classification Benchmark]{}. In: CoRR (2016)

Alletto, S., Serra, G., Calderara, S., Cucchiara, R.: Understanding social relationships in egocentric vision. In: [Pattern Recognition]{} (2015)

Antol, S., Agrawal, A., Lu, J., Mitchell, M., Batra, D., Zitnick, C.L., Parikh, D.: [VQA]{}: [V]{}isual [Q]{}uestion [A]{}nswering. In: ICCV (2015)

Banerjee, S., Pedersen, T.: An adapted lesk algorithm for word sense disambiguation using wordnet. In: [CICLing]{} (2002)

: [CMU]{} sphinx. https://cmusphinx.github.io/

Damen, D., Leelasawassuk, T., Haines, O., Calway, A., Mayol-Cuevas, W.: You-do, [I]{}-learn: Discovering task relevant objects and their modes of interaction from multi-user egocentric video. In: BMVC (2014)

Das, A., Kottur, S., Gupta, K., Singh, A., Yadav, D., Moura, J.M., Parikh, D., Batra, D.: [V]{}isual [D]{}ialog. In: CVPR (2017)

De La Torre, F., Hodgins, J., Bargteil, A., Martin, X., Macey, J., Collado, A., Beltran, P.: Guide to the [C]{}arnegie [M]{}ellon [U]{}niversity [M]{}ultimodal [A]{}ctivity ([CMU-MMAC]{}) database. In: Robotics Institute (2008)

Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: Imagenet: A large-scale hierarchical image database. In: CVPR (2009)

Doughty, H., Damen, D., Mayol-Cuevas, W.: Who’s better? who’s best? pairwise deep ranking for skill determination. In: CVPR (2018)

Everingham, M., Van Gool, L., Williams, C.K.I., Winn, J., Zisserman, A.: The [PASCAL]{} [V]{}isual [O]{}bject [C]{}lasses [(VOC)]{} [C]{}hallenge. In: IJCV (2010)

Fathi, A., Hodgins, J., Rehg, J.: Social interactions: A first-person perspective. In: CVPR (2012)

Fathi, A., Li, Y., Rehg, J.: Learning to recognize daily actions using gaze. In: ECCV (2012)

Fouhey, D.F., Kuo, W.c., Efros, A.A., Malik, J.: From lifestyle vlogs to everyday interactions. arXiv preprint arXiv:1712.02310 (2017)

Furnari, A., Battiato, S., Grauman, K., Farinella, G.M.: Next-active-object prediction from egocentric videos. In: JVCIR (2017)

: [Extended GTEA Gaze+]{}. http://webshare.ipat.gatech.edu/coc-rim-wall-lab/web/yli440/egtea\_gp (2018)

Google: Google cloud speech api. https://cloud.google.com/speech

Goyal, R., Kahou, S.E., Michalski, V., Materzynska, J., Westphal, S., Kim, H., Haenel, V., Fr[ü]{}nd, I., Yianilos, P., Mueller[-]{}Freitag, M., Hoppe, F., Thurau, C., Bax, I., Memisevic, R.: The “something something” video database for learning and evaluating visual common sense. In: ICCV (2017)

He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: CVPR (2016)

Heidarivincheh, F., Mirmehdi, M., Damen, D.: Action completion: A temporal model for moment detection. In: BMVC (2018)

Huang, J., Rathod, V., Chow, D., Sun, C., Zhu, M., Fathi, A., Lu, Z.: [Tensorflow Object Detection API]{}. https://github.com/tensorflow/models/tree/master/research/object\_detection

Huang, J., Rathod, V., Sun, C., Zhu, M., Korattikara, A., Fathi, A., Fischer, I., Wojna, Z., Song, Y., Guadarrama, S., et al.: Speed/accuracy trade-offs for modern convolutional object detectors. In: CVPR (2017)

IBM: [IBM]{} watson speech to text. https://www.ibm.com/watson/services/speech-to-text

Ioffe, S., Szegedy, C.: Batch normalization: Accelerating deep network training by reducing internal covariate shift. In: ICML (2015)

Kalogeiton, V., Weinzaepfel, P., Ferrari, V., Schmid, C.: [Joint learning of object and action detectors]{}. In: [ICCV]{} (2017)

Karpathy, A., Fei-Fei, L.: [Deep Visual-Semantic Alignments for Generating Image Descriptions]{}. In: CVPR (2015)

Krizhevsky, A., Sutskever, I., Hinton, G.E.: [Imagenet classification with deep convolutional neural networks]{}. In: NIPS (2012)

Kuehne, H., Arslan, A., Serre, T.: [The Language of Actions: Recovering the Syntax and Semantics of Goal-Directed Human Activities]{}. In: CVPR (2014)

Lee, Y., Ghosh, J., Grauman, K.: Discovering important people and objects for egocentric video summarization. In: CVPR (2012)

Lin, T.Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Doll[á]{}r, P., Zitnick, C.L.: Microsoft [COCO]{}: Common objects in context. In: ECCV (2014)

Mikolov, T., Chen, K., Corrado, G., Dean, J.: Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781 (2013)

Miller, G.: Wordnet: a lexical database for english. In: CACM (1995)

Moltisanti, D., Wray, M., Mayol-Cuevas, W., Damen, D.: Trespassing the boundaries: Labeling temporal bounds for object interactions in egocentric video. In: ICCV (2017)

Nair, A., Chen, D., Agrawal, P., Isola, P., Abbeel, P., Malik, J., Levine, S.: Combining self-supervised learning and imitation for vision-based rope manipulation. In: ICRA (2017)

Park, H.S., Hwang, J.J., Niu, Y., Shi, J.: Egocentric future localization. In: CVPR (2016)

Pirsiavash, H., Ramanan, D.: Detecting activities of daily living in first-person camera views. In: CVPR (2012)

Ren, S., He, K., Girshick, R., Sun, J.: Faster [R-CNN]{}: Towards real-time object detection with region proposal networks. In: NIPS (2015)

Rohrbach, A., Rohrbach, M., Tandon, N., Schiele, B.: [A Dataset for Movie Description]{}. In: CVPR (2015)

Rohrbach, M., Amin, S., Andriluka, M., Schiele, B.: [A Database for Fine Grained Activity Detection of Cooking Activities]{}. In: CVPR (2012)

Ryoo, M.S., Matthies, L.: First-person activity recognition: What are they doing to me? In: CVPR (2013)

Sigurdsson, G.A., Gupta, A., Schmid, C., Farhadi, A., Alahari, K.: Charades-ego: A large-scale dataset of paired third and first person videos. In: ArXiv (2018)

Sigurdsson, G.A., Varol, G., Wang, X., Farhadi, A., Laptev, I., Gupta, A.: Hollywood in homes: Crowdsourcing data collection for activity understanding. In: ECCV (2016)

Simonyan, K., Zisserman, A.: Two-stream convolutional networks for action recognition in videos. In: Advances in neural information processing systems. pp. 568–576 (2014)

Stein, S., McKenna, S.: [Combining Embedded Accelerometers with Computer Vision for Recognizing Food Preparation Activities]{}. In: [UbiComp]{} (2013)

Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., Rabinovich, A.: Going deeper with convolutions. In: CVPR (2015)

Tapaswi, M., Zhu, Y., Stiefelhagen, R., Torralba, A., Urtasun, R., Fidler, S.: [MovieQA]{}: Understanding stories in movies through question-answering. In: CVPR (2016)

Vondrick, C., Pirsiavash, H., Torralba, A.: Anticipating visual representations from unlabeled video. In: CVPR (2016)

Wang, L., Xiong, Y., Wang, Z., Qiao, Y., Lin, D., Tang, X., [Val Gool]{}, L.: Temporal segment networks: Towards good practices for deep action recognition. In: ECCV (2016)

Yamaguchi, K.: [Bbox-annotator]{}. https://github.com/kyamagu/bbox-annotator

Yeung, S., Russakovsky, O., Jin, N., Andriluka, M., Mori, G., Fei-Fei, L.: Every moment counts: Dense detailed labeling of actions in complex videos. IJCV (2018)

Yuanjun, X.: [PyTorch]{} [Temporal Segment Network]{}. https://github.com/yjxiong/tsn-pytorch (2017)

Zach, C., Pock, T., Bischof, H.: A duality based approach for realtime [TV-L1]{} optical flow. In: Pattern Recognition (2007)

Zhang, T., McCarthy, Z., Jow, O., Lee, D., Goldberg, K., Abbeel, P.: Deep imitation learning for complex manipulation tasks from virtual reality teleoperation. In: ICRA (2018)

Zhao, H., Yan, Z., Wang, H., Torresani, L., Torralba, A.: [SLAC: A Sparsely Labeled Dataset for Action Classification and Localization]{}. arXiv preprint arXiv:1712.09374 (2017)

Zhou, B., Zhao, H., Puig, X., Fidler, S., Barriuso, A., Torralba, A.: Scene parsing through ade20k dataset. In: CVPR (2017)

Zhou, L., Xu, C., Corso, J.J.: Towards automatic learning of procedures from web instructional videos. arXiv preprint arXiv:1703.09788 (2017)

[^1]:
