  The $N$-dimensional Smorodinsky-Winternitz system is a maximally
superintegrable and exactly solvable model, being subject of study from
different approaches. The model has been demonstrated to be multiseparable with
wavefunctions given by Laguerre and Jacobi polynomials. In this paper we
present the complete symmetry algebra ${\cal SW}(N)$ of the system, which it is
a higher-rank quadratic one containing the recently discovered Racah algebra
${\cal R}(N)$ as subalgebra. The substructures of distinct quadratic ${\cal
Q}(3)$ algebras and their related Casimirs are also studied. In this way, from
the constraints on the oscillator realizations of these substructures, the
energy spectrum of the $N$-dimensional Smorodinsky-Winternitz system is
obtained. We show that ${\cal SW}(N)$ allows different set of substructures
based on the Racah algebra ${\cal R}({ N})$ which can be applied independently
to algebraically derive the spectrum of the system.
