  Orthogonal time frequency space (OTFS) is a promising alternative to
orthogonal frequency division multiplexing (OFDM) for high-mobility
communications. We propose a novel multiple-input multiple-output (MIMO)
integrated sensing and communication (ISAC) system based on OTFS modulation. We
begin by deriving new sensing and communication signal models for the proposed
MIMO-OTFS ISAC system that explicitly capture inter-symbol interference (ISI)
and inter-carrier interference (ICI) effects. We then develop a generalized
likelihood ratio test (GLRT) based multi-target detection and
delay-Doppler-angle estimation algorithm for MIMO-OTFS radar sensing that can
simultaneously mitigate and exploit ISI/ICI effects, to prevent target masking
and surpass standard unambiguous detection limits in range/velocity. Moreover,
considering two operational modes (search/track), we propose an adaptive
MIMO-OTFS ISAC transmission strategy. For the search mode, we introduce the
concept of delay-Doppler (DD) multiplexing, enabling omnidirectional probing of
the environment and large virtual array at the OTFS radar receiver. For the
track mode, we pursue a directional transmission approach and design an OTFS
ISAC optimization algorithm in spatial and DD domains, seeking the optimal
trade-off between radar signal-to-noise ratio (SNR) and achievable rate.
Simulation results verify the effectiveness of the proposed sensing algorithm
and reveal valuable insights into OTFS ISAC trade-offs under varying
communication channel characteristics.
