  We study the potential employment of improper Gaussian signaling (IGS) in
full-duplex relaying (FDR) with non-negligible residual self-interference (RSI)
under Nakagami- m fading. IGS is recently shown to outperform traditional
proper Gaussian signaling (PGS) in several interference-limited settings. In
this work, IGS is employed as an attempt to alleviate RSI. We use two
performance metrics, namely, the outage probability and the ergodic rate.
First, we provide upper and lower bounds for the system performance in terms of
the relay transmit power and circularity coefficient, a measure of the signal
impropriety. Then, we numerically optimize the relay signal parameters based
only on the channel statistics to improve the system performance. Based on the
analysis, IGS allows FDR to operate even with high RSI. The results show that
IGS can leverage higher power budgets to enhance the performance, meanwhile it
relieves RSI impact via tuning the signal impropriety. Interestingly,
one-dimensional optimization of the circularity coefficient, with maximum relay
power, offers a similar performance as the joint optimization, which reduces
the optimization complexity. From a throughput standpoint, it is shown that
IGS-FDR can outperform not only PGS-FDR, but also half-duplex relaying
with/without maximum ratio combining over certain regions of the target source
rate.
