  Ion wake induced mode coupling is investigated experimentally for a
horizontal dust chain formed in a complex plasma, verifying results from
previous simulation. A double branch of faint spectral lines is detected in the
mode spectra which verifies the predicted rule of mode coupling between the
vertical z(j=i$\pm$1) modes and the longitudinal mode x(i). Discreet
instabilities are observed as the branches of x- and z-modes intersect each
other. The mode spectra in the vicinity of the instabilities exhibit enhanced
energy density at specific coupled x and z modes, serving as direct evidence
that these instabilities are caused by resonance between the coupled modes. The
instability-induced melting threshold was found to obey the Lindemann criterion
through analysis of the instantaneous relative interparticle distance
fluctuation (IDF). The relation between mode spectra and dispersion relations
was further studied by multiplying the mode spectra with a transition matrix
connecting the bases of normal mode eigenvectors and Fourier series in k space.
Typical dispersion relations corresponding to the longitudinal and out-of-plane
transverse Dust Lattice Waves (DLWs) are obtained, which also exhibit
characteristics unique to finite systems, including discrete bands and strong
fluctuations in the energy density.
