  Projected Entangled Pair States (PEPS) provide a framework for the
construction of models where a single tensor gives rise to both Hamiltonian and
ground state wavefunction on the same footing. A key problem is to characterize
the behavior which emerges in the system in terms of the properties of the
tensor, and thus of the Hamiltonian. In this paper, we consider PEPS models
with $\mathbb Z_2$ on-site symmetry and study the occurence of long-range order
and spontaneous symmetry breaking. We show how long-range order is connected to
a degeneracy in the spectrum of the PEPS transfer operator, and how the latter
gives rise to spontaneous symmetry breaking under perturbations. We provide a
succinct characterization of the symmetry broken states in terms of the PEPS
tensor, and find that using the symmetry broken states we can derive a local
entanglement Hamiltonian, thereby restoring locality of the entanglement
Hamiltonian for all gapped phases.
