  The Hofstadter model, well-known for its fractal butterfly spectrum,
describes two-dimensional electrons under a perpendicular magnetic field, which
gives rise to the integer quantum hall effect. Inspired by the real-space
building blocks of non-Abelian gauge fields from a recent experiment [Science,
365, 1021 (2019)], we introduce and theoretically study two non-Abelian
generalizations of the Hofstadter model. Each model describes two pairs of
Hofstadter butterflies that are spin-orbit coupled. In contrast to the original
Hofstadter model that can be equivalently studied in the Landau and symmetric
gauges, the corresponding non-Abelian generalizations exhibit distinct spectra
due to the non-commutativity of the gauge fields. We derive the genuine
(necessary and sufficient) non-Abelian condition for the two models from the
commutativity of their arbitrary loop operators. At zero energy, the models are
gapless and host Weyl and Dirac points protected by internal and crystalline
symmetries. Double (8-fold), triple (12-fold), and quadrupole (16-fold) Dirac
points also emerge, especially under equal hopping phases of the non-Abelian
potentials. At other fillings, the gapped phases of the models give rise to
$\mathbb{Z}_2$ topological insulators. We conclude by discussing possible
schemes for the experimental realizations of the models in photonic platforms.
