  This paper focuses on the signal design in a Gaussian amplify-and-forward
(AF) relay system with superposition coding (SC) being applied at the relay,
which allows the source and relay to transmit their own information within two
time slots. Practical quadrature amplitude modulation (QAM) constellations are
adopted at the source and relay. To improve the system error performance, we
optimize the weight coefficients adopted at the source and relay to maximize
the minimum Euclidean distance of the received composite constellation, subject
to their individual average power constraints. The formulated optimization
problem is shown to be a mixed continuous-discrete one that is non-trivial to
resolve in general. By resorting to the punched Farey sequence, we manage to
obtain the optimal solution to the formulated problem by first partitioning the
entire feasible region into a finite number of sub-intervals and then taking
the maximum over all the possible sub-intervals. Simulation results are
provided to demonstrate the superior performance of our proposed design based
on SC over that using conventional time division multiple access (TDMA).
