  Full-duplex transmission comprises the ability to transmit and receive at the
same time on the same frequency band. It allows for more efficient utilization
of spectral resources, but raises the challenge of strong self-interference
(SI). Cancellation of SI is generally implemented as a multi-stage approach.
This work proposes a novel adaptive SI cancellation algorithm in the digital
domain and a comprehensive analysis of state-of-the-art adaptive cancellation
techniques. Inspired by recent progress in acoustic echo control, we introduce
a composite state-space model of the nonlinear SI channel in cascade structure.
We derive a SI cancellation algorithm that decouples the identification of
linear and nonlinear elements of the composite state. They are estimated
separately and consecutively in each adaptation cycle by a Kalman filter in DFT
domain. We show that this adaptation can be supported by a-priori signal
orthogonalization and decoding of the signal-of-interest (SoI). In our
simulation results, we analyze the performance by evaluating residual
interference, system identification accuracy and communication rate. Based on
the results, we provide recommendations for system design. In case of input
orthogonalization, our Kalman filter solution in cascade structure delivers
best performance with low computational complexity. In this configuration, the
performance lines up with that of the monolithic (parallel) Kalman filter or
the recursive-least squares (RLS) algorithms. We show that the Kalman-based
algorithm is superior over the RLS under time-variant conditions if the SoI is
decoded and in this way the covariance information required by the Kalman
filter can be provided to it.
