  We consider the problem of minimizing a non-convex function over a smooth
manifold $\mathcal{M}$. We propose a novel algorithm, the Orthogonal Directions
Constrained Gradient Method (ODCGM) which only requires computing a projection
onto a vector space. ODCGM is infeasible but the iterates are constantly pulled
towards the manifold, ensuring the convergence of ODCGM towards $\mathcal{M}$.
ODCGM is much simpler to implement than the classical methods which require the
computation of a retraction. Moreover, we show that ODCGM exhibits the
near-optimal oracle complexities $\mathcal{O}(1/\varepsilon^2)$ and
$\mathcal{O}(1/\varepsilon^4)$ in the deterministic and stochastic cases,
respectively. Furthermore, we establish that, under an appropriate choice of
the projection metric, our method recovers the landing algorithm of Ablin and
Peyr\'e (2022), a recently introduced algorithm for optimization over the
Stiefel manifold. As a result, we significantly extend the analysis of Ablin
and Peyr\'e (2022), establishing near-optimal rates both in deterministic and
stochastic frameworks. Finally, we perform numerical experiments which shows
the efficiency of ODCGM in a high-dimensional setting.
