  Measurements of the elliptic flow coefficient relative to the collision plane
defined by the spectator neutrons $v_2${$\Psi_{\rm SP}$} in collisions of Pb
ions at center-of-mass energy per nucleon-nucleon pair $\sqrt{s_{\rm NN}}$=2.76
TeV and Xe ions at $\sqrt{s_{\rm NN}}$=5.44 TeV are reported. The results are
presented for charged particles produced at midrapidity as a function of
centrality and transverse momentum. The ratio between $v_2${$\Psi_{\rm SP}$}
and the elliptic flow coefficient relative to the participant plane $v_2${4},
estimated using four-particle correlations, deviates by up to 20% from unity
depending on centrality. This observation differs strongly from the magnitude
of the corresponding eccentricity ratios predicted by the TRENTo and the
elliptic power models of initial state fluctuations that are tuned to describe
the participant plane anisotropies. The differences can be interpreted as a
decorrelation of the neutron spectator plane and the reaction plane because of
fragmentation of the remnants from the colliding nuclei, which points to an
incompleteness of current models of initial state fluctuations. A significant
transverse momentum dependence of the ratio $v_2${$\Psi_{\rm SP}$}/$v_2${4} is
observed in all but the most central collisions, which may help to understand
whether momentum anisotropies at low and intermediate transverse momentum have
a common origin in initial state fluctuations. The ratios of $v_2${$\Psi_{\rm
SP}$} and $v_2${4} to the corresponding initial state eccentricities for Xe-Xe
and Pb-Pb collisions at similar initial entropy density show a difference of
$(7.0 \pm 0.9)$% with an additional variation of +1.8% when including RHIC data
in the TRENTo parameter extraction. These observations provide new experimental
constraints for viscous effects in the hydrodynamic modeling of the expanding
quark-gluon plasma.
