  We explore the problem of automatically inferring the amount of kilocalories
used by human during physical activity from his/her video observation. To study
this underresearched task, we introduce Vid2Burn -- an omni-source benchmark
for estimating caloric expenditure from video data featuring both, high- and
low-intensity activities for which we derive energy expenditure annotations
based on models established in medical literature. In practice, a training set
would only cover a certain amount of activity types, and it is important to
validate, if the model indeed captures the essence of energy expenditure,
(e.g., how many and which muscles are involved and how intense they work)
instead of memorizing fixed values of specific activity categories seen during
training. Ideally, the models should look beyond such category-specific biases
and regress the caloric cost in videos depicting activity categories not
explicitly present during training. With this property in mind, Vid2Burn is
accompanied with a cross-category benchmark, where the task is to regress
caloric expenditure for types of physical activities not present during
training. An extensive evaluation of state-of-the-art approaches for video
recognition modified for the energy expenditure estimation task demonstrates
the difficulty of this problem, especially for new activity types at test-time,
marking a new research direction. Dataset and code are available at
https://github.com/KPeng9510/Vid2Burn.
