  Vector approximate message passing (VAMP) is an efficient approximate
inference algorithm used for generalized linear models. Although VAMP exhibits
excellent performance, particularly when measurement matrices are sampled from
rotationally invariant ensembles, existing convergence and performance analyses
have been limited mostly to cases in which the correct posterior distribution
is available. Here, we extend the analyses for cases in which the correct
posterior distribution is not used in the inference stage. We derive state
evolution equations, which macroscopically describe the dynamics of VAMP, and
show that their fixed point is consistent with the replica symmetric solution
obtained by the replica method of statistical mechanics. We also show that the
fixed point of VAMP can exhibit a microscopic instability, the critical
condition of which agrees with that for breaking the replica symmetry. The
results of numerical experiments support our findings.
