  The problem of diagonalizing a class of complicated matrices, to be called
ultrametric matrices, is investigated. These matrices appear at various stages
in the description of disordered systems with many equilibrium phases by the
technique of replica symmetry breaking. The residual symmetry, remaining after
the breaking of permutation symmetry between replicas, allows us to bring all
ultrametric matrices to a block diagonal form by a common similarity
transformation. A large number of these blocks are, in fact, of size 1 times 1,
i.e. in a vast sector the transformation actually diagonalizes the matrix. In
the other sectors we end up with blocks of size (R+1) times (R+1) where R is
the number of replica symmetry breaking steps. These blocks cannot be further
reduced without giving more information, in addition to ultrametric symmetry,
about the matrix. Similar results for the inverse of a generic ultrametric
matrix are also derived.
