  The paper presents a novel type of capsule network (CAP) that uses
custom-defined neural network (NN) layers for blind classification of digitally
modulated signals using their in-phase/quadrature (I/Q) components. The custom
NN layers of the CAP are inspired by cyclostationary signal processing (CSP)
techniques and implement feature extraction capabilities that are akin to the
calculation of the cyclic cumulant (CC) features employed in conventional
CSP-based approaches to blind modulation classification and signal
identification. The classification performance and the generalization abilities
of the proposed CAP are tested using two distinct datasets that contain similar
classes of digitally modulated signals but that have been generated
independently, and numerical results obtained reveal that the proposed CAP with
novel NN feature extraction layers achieves high classification accuracy while
also outperforming alternative deep learning (DL)-based approaches for signal
classification in terms of both classification accuracy and generalization
abilities.
