  Detecting human-object interactions (HOI) is an important step toward a
comprehensive visual understanding of machines. While detecting non-temporal
HOIs (e.g., sitting on a chair) from static images is feasible, it is unlikely
even for humans to guess temporal-related HOIs (e.g., opening/closing a door)
from a single video frame, where the neighboring frames play an essential role.
However, conventional HOI methods operating on only static images have been
used to predict temporal-related interactions, which is essentially guessing
without temporal contexts and may lead to sub-optimal performance. In this
paper, we bridge this gap by detecting video-based HOIs with explicit temporal
information. We first show that a naive temporal-aware variant of a common
action detection baseline does not work on video-based HOIs due to a
feature-inconsistency issue. We then propose a simple yet effective
architecture named Spatial-Temporal HOI Detection (ST-HOI) utilizing temporal
information such as human and object trajectories, correctly-localized visual
features, and spatial-temporal masking pose features. We construct a new video
HOI benchmark dubbed VidHOI where our proposed approach serves as a solid
baseline.
