  This paper studies the cooperation between a primary system and a cognitive
system in a cellular network where the cognitive base station (CBS) relays the
primary signal using amplify-and-forward or decode-and-forward protocols, and
in return it can transmit its own cognitive signal. While the commonly used
half-duplex (HD) assumption may render the cooperation less efficient due to
the two orthogonal channel phases employed, we propose that the CBS can work in
a full-duplex (FD) mode to improve the system rate region. The problem of
interest is to find the achievable primary-cognitive rate region by studying
the cognitive rate maximization problem. For both modes, we explicitly consider
the CBS transmit imperfections, which lead to the residual self-interference
associated with the FD operation mode. We propose closed-form solutions or
efficient algorithms to solve the problem when the related residual
interference power is non-scalable or scalable with the transmit power.
Furthermore, we propose a simple hybrid scheme to select the HD or FD mode
based on zero-forcing criterion, and provide insights on the impact of system
parameters. Numerical results illustrate significant performance improvement by
using the FD mode and the hybrid scheme.
