  The orbital angular momentum (OAM) wireless communication technique is
emerging as one of potential techniques for the Sixth generation (6G) wireless
communication system. The most advantage of OAM wireless communication
technique is the natural orthogonality among different OAM states. However, one
of the most disadvantages is the crosstalk among different OAM states which is
widely caused by the atmospheric turbulence and misalignment between
transmitting and receiving antennas. Considering the OAM-based multiple-input
multiple-output (OAM-MIMO) transmission system with unaligned antennas, a new
channel model is proposed for performance analysis. Moreover, a purity model of
the OAM-MIMO transmission system with unaligned antennas is derived for the
non-Kolmogorov turbulence. Furthermore, error probability and capacity models
are derived for OAM-MIMO transmission systems with unaligned antennas. To
overcome the disadvantage caused by unaligned antennas and non-Kolmogorov
turbulence, a new optimization algorithm of OAM state interval is proposed to
improve the capacity of OAM-MIMO transmission system. Numerical results
indicate that the capacity of OAM-MIMO transmission system is improved by the
optimization algorithm. Specifically, the capacity increment of OAM-MIMO
transmission system adopting the optimization algorithm is up to 28.7% and
320.3% when the angle of deflection between transmitting and receiving antennas
is -24 dB and -5 dB, respectively.
