  We show how to perform exact diagonalizations of $\mathrm{SU}(N)$
Fermi-Hubbard models on $L$-site clusters separately in each irreducible
representation ({irrep}) of $\mathrm{SU}(N)$. Using the representation theory
of the unitary group $\mathrm{U}(L)$, we demonstrate that a convenient
orthonormal basis, on which matrix elements of the Hamiltonian are very simple,
is given by the set of {\it semistandard Young tableaux} (or, equivalently the
Gelfand-Tsetlin patterns) corresponding to the targeted irrep. As an
application of this color factorization, we study the robustness of some
$\mathrm{SU}(N)$ phases predicted in the Heisenberg limit upon decreasing the
on-site interaction $U$ on various lattices of size $L \leq 12$ and for $2 \leq
N \leq 6$. In particular, we show that a long-range color ordered phase emerges
for intermediate $U$ for $N=4$ at filling $1/4$ on the triangular lattice.
