  Most benchmarks for studying surgical interventions focus on a specific
challenge instead of leveraging the intrinsic complementarity among different
tasks. In this work, we present a new experimental framework towards holistic
surgical scene understanding. First, we introduce the Phase, Step, Instrument,
and Atomic Visual Action recognition (PSI-AVA) Dataset. PSI-AVA includes
annotations for both long-term (Phase and Step recognition) and short-term
reasoning (Instrument detection and novel Atomic Action recognition) in
robot-assisted radical prostatectomy videos. Second, we present Transformers
for Action, Phase, Instrument, and steps Recognition (TAPIR) as a strong
baseline for surgical scene understanding. TAPIR leverages our dataset's
multi-level annotations as it benefits from the learned representation on the
instrument detection task to improve its classification capacity. Our
experimental results in both PSI-AVA and other publicly available databases
demonstrate the adequacy of our framework to spur future research on holistic
surgical scene understanding.
