  When trying to fit a deep neural network (DNN) to a $G$-invariant target
function with $G$ a group, it only makes sense to constrain the DNN to be
$G$-invariant as well. However, there can be many different ways to do this,
thus raising the problem of ``$G$-invariant neural architecture design'': What
is the optimal $G$-invariant architecture for a given problem? Before we can
consider the optimization problem itself, we must understand the search space,
the architectures in it, and how they relate to one another. In this paper, we
take a first step towards this goal; we prove a theorem that gives a
classification of all $G$-invariant single-hidden-layer or ``shallow'' neural
network ($G$-SNN) architectures with ReLU activation for any finite orthogonal
group $G$, and we prove a second theorem that characterizes the inclusion maps
or ``network morphisms'' between the architectures that can be leveraged during
neural architecture search (NAS). The proof is based on a correspondence of
every $G$-SNN to a signed permutation representation of $G$ acting on the
hidden neurons; the classification is equivalently given in terms of the first
cohomology classes of $G$, thus admitting a topological interpretation. The
$G$-SNN architectures corresponding to nontrivial cohomology classes have, to
our knowledge, never been explicitly identified in the literature previously.
Using a code implementation, we enumerate the $G$-SNN architectures for some
example groups $G$ and visualize their structure. Finally, we prove that
architectures corresponding to inequivalent cohomology classes coincide in
function space only when their weight matrices are zero, and we discuss the
implications of this for NAS.
