I am puzzled by the amazing utility and therefore ubiquity of
two-dimensional matrices in comparison to the relative
paucity of multidimensional arrays of numbers, hypermatrices.
Of course multidimensional arrays are useful:
every programming language supports them, and I often employ them myself.
But these uses treat the arrays primarily as convenient data structures
rather than as mathematical objects.
When I think of the generalization of polygon to $d$-dimensional polytope,
or of two-dimensional surface to $n$-dimensional manifold, 
I see an increase in mathematical importance and utility;
whereas with matrices, the opposite.

One answer to my question that I am prepared to acknowledge is that
my perception is clouded by ignorance:
hypermatrices are just as important, useful, and prevalent in mathematics
as 2D matrices.
Perhaps tensors, especially when viewed as multilinear maps, fulfill
this role.  Certainly they play a crucial role in physics,
fluid mechanics, Riemannian geometry, and other areas.
Perhaps there is a rich spectral theory 
of hypermatrices, a rich decomposition (LU, QR, Cholesky, etc.) theory
of hypermatrices, a rich theory of random hypermatrices—all
analogous to corresponding theories of 2D matrices, all of which I am unaware.

I do know that Cayley explored hyperdeterminants in the 19th century, and that
Gelfand, Kapranov, and Zelevinsky wrote a book
entitled Discriminants, Resultants and Multidimensional Determinants
(Birkhäuser, Boston, 1994) about which I know little.

If, despite my ignorance, indeed hypermatrices have found only relatively
rare utility in mathematics, I would be interested to know if there
is some high-level reason for this, some reason that 2D matrices
are inherently more useful than hypermatrices?

I am aware of how amorphous is this question, and apologize if it is considered inappropriate.

Note that in linear algebra matrices describe at least two different things: linear maps between vector spaces (we consider only finite-dimensional vector spaces here) and bilinear forms. When thinking of matrices as tensors, linear maps between $V$ and $W$ are elements of the space $V^* \otimes W$, whereas bilinear forms between $V$ and $W$ are elements of $V^* \otimes W^* $. Now you can easily generalize the latter case to more than two spaces, but not the former. But it is the former case where several concepts like composition (matrix multiplication), determinants, eigenvalues etc. apply. (Note that eigenvalues and determinants can be defined for bilinear forms on a vector space equipped with an inner product, but not for bilinear forms on plain vector spaces). Of course you can consider spaces like $V^* \otimes W^* \otimes X$, but elements of this space are better thought as linear maps between $V\otimes W$ and $X$ than as three-dimensional hypermatrices. So what is special about the number 2 is that there is a notion of duality for vector spaces, but no "n-ality".