  This paper investigates the average symbol error rate (ASER) performance of a
dual-hop hybrid relaying system relying on both radio frequency (RF) and free
space optical (FSO) links. Specifically, the RF link is used for supporting
mobile communication, while the FSO link is adopted as the backhaul of the
cellular infrastructure. Considering non-line-of-sight (NLoS) RF transmissions
and a generalized atmospheric turbulence (AT) channel, the associated
statistical features constituted of both the exact and the asymptotic moment
generating functions (MGF) are derived in closed form. They are then used for
calculating the ASER of M-ary phase shift keying (PSK), differentially encoded
non-coherent PSK (DPSK) and non-coherent frequency-shift keying (FSK). A range
of additional asymptotic expressions are also derived for all the modulation
schemes under high signal-to-noise ratios (SNR). It is observed from the
asymptotic analysis that the ASERs of all the modulation schemes are dominated
by the average SNR of the RF link in the hybrid relaying system using a fixed
relay gain, while in the relaying system using a dynamic channel dependent
relay gain, the ASERs of all the modulation schemes depend both on the average
SNR and on the AT condition of the FSO path. We also find that the fixed-gain
relaying strategy achieves twice the diversity order of the channel-dependent
relaying strategy albeit at the cost of requiring a high power amplifier (PA)
dynamic range at the relay node. Furthermore, by comparing the asymptotic
ASERs, we calculate the SNR differences between the different modulation
schemes in both the fixed-gain and the channel-dependent relaying system.
Finally, simulation results are presented for confirming the accuracy of our
expressions and observations.
