  Despite continual learning's long and well-established academic history, its
application in real-world scenarios remains rather limited. This paper contends
that this gap is attributable to a misalignment between the actual challenges
of continual learning and the evaluation protocols in use, rendering proposed
solutions ineffective for addressing the complexities of real-world setups. We
validate our hypothesis and assess progress to date, using a new 3D semantic
segmentation benchmark, OCL-3DSS. We investigate various continual learning
schemes from the literature by utilizing more realistic protocols that
necessitate online and continual learning for dynamic, real-world scenarios
(eg., in robotics and 3D vision applications). The outcomes are sobering: all
considered methods perform poorly, significantly deviating from the upper bound
of joint offline training. This raises questions about the applicability of
existing methods in realistic settings. Our paper aims to initiate a paradigm
shift, advocating for the adoption of continual learning methods through new
experimental protocols that better emulate real-world conditions to facilitate
breakthroughs in the field.
