  The design of message passing (MP) algorithms on factor graphs is an
effective manner to implement channel estimation (CE) in wireless communication
systems, which performance can be further improved by exploiting prior
probability models that accurately match the channel characteristics. In this
work, we study the CE problem in a downlink massive multiple-input
multiple-output (MIMO) orthogonal frequency division multiplexing (OFDM)
system. As the prior probability, we propose the Markov chain two-state
Gaussian mixture with large variance differences (TSGM-LVD) model to exploit
the structured sparsity in the angle-frequency domain of the channel. Existing
single and combined MP rules cannot deal with the message computation of the
proposed probability model. To overcome this issue, we present a general method
to derive the hybrid message passing (HMP) rule, which allows the calculation
of messages described by mixed linear and non-linear functions. Accordingly, we
design the HMP-TSGM-LVD algorithm under the structured turbo framework (STF).
Simulation results demonstrate that the proposed algorithm converges faster and
obtains better and more stable performance than its counterparts. In
particular, the gain of the proposed approach is maximum (3 dB) in the high
signal-to-noise ratio regime, while benchmark approaches experience oscillating
behavior due to the improper prior model characterization.
