  Continual learning aims to avoid catastrophic forgetting and effectively
leverage learned experiences to master new knowledge. Existing gradient
projection approaches impose hard constraints on the optimization space for new
tasks to minimize interference, which simultaneously hinders forward knowledge
transfer. To address this issue, recent methods reuse frozen parameters with a
growing network, resulting in high computational costs. Thus, it remains a
challenge whether we can improve forward knowledge transfer for gradient
projection approaches using a fixed network architecture. In this work, we
propose the Restricted Orthogonal Gradient prOjection (ROGO) framework. The
basic idea is to adopt a restricted orthogonal constraint allowing parameters
optimized in the direction oblique to the whole frozen space to facilitate
forward knowledge transfer while consolidating previous knowledge. Our
framework requires neither data buffers nor extra parameters. Extensive
experiments have demonstrated the superiority of our framework over several
strong baselines. We also provide theoretical guarantees for our relaxing
strategy.
