  Permutation invariant polynomial functions of matrices have previously been
studied as the observables in matrix models invariant under $S_N$, the
symmetric group of all permutations of $N$ objects. In this paper, the
permutation invariant matrix observables (PIMOs) of degree $k$ are shown to be
in one-to-one correspondence with equivalence classes of elements in the
diagrammatic partition algebra $P_k(N)$. On a 4-dimensional subspace of the
13-parameter space of $S_N$ invariant Gaussian models, there is an enhanced
$O(N)$ symmetry. At a special point in this subspace, is the simplest $O(N)$
invariant action. This is used to define an inner product on the PIMOs which is
expressible as a trace of a product of elements in the partition algebra. The
diagram algebra $P_k(N)$ is used to prove the large $N$ factorisation property
of this inner product, which generalizes a familiar large $N$ factorisation for
inner products of matrix traces invariant under continuous symmetries.
