  Higher order Laguerre-Gauss (LG) beams have been proposed for use in future
gravitational wave detectors, such as upgrades to the Advanced LIGO detectors
and the Einstein Telescope, for their potential to reduce the effects of the
thermal noise of the test masses. This paper details the theoretical analysis
and simulation work carried out to investigate the behaviour of LG beams in
realistic optical setups, in particular the coupling between different LG modes
in a linear cavity. We present a new analytical approximation to compute the
coupling between modes, using Zernike polynomials to describe mirror surface
distortions. We apply this method in a study of the behaviour of the LG33 mode
within realistic arm cavities, using measured mirror surface maps from the
Advanced LIGO project. We show mode distortions that can be expected to arise
due to the degeneracy of higher order spatial modes within such cavities and
relate this to the theoretical analysis. Finally we identify the mirror
distortions which cause significant coupling from the LG33 mode into other
order 9 modes and derive requirements for the mirror surfaces.
