  Internal Language Model (LM)-based methods use permutation language modeling
(PLM) to solve the error correction caused by conditional independence in
external LM-based methods. However, random permutations of human interference
cause fit oscillations in the model training, and Iterative Refinement (IR)
operation to improve multimodal information decoupling also introduces
additional overhead. To address these issues, this paper proposes the
Hierarchical Attention autoregressive Model with Adaptive Permutation (HAAP) to
enhance the location-context-image interaction capability, improving
autoregressive generalization with internal LM. First, we propose Implicit
Permutation Neurons (IPN) to generate adaptive attention masks to dynamically
exploit token dependencies. The adaptive masks increase the diversity of
training data and prevent model dependency on a specific order. It reduces the
training overhead of PLM while avoiding training fit oscillations. Second, we
develop Cross-modal Hierarchical Attention mechanism (CHA) to couple context
and image features. This processing establishes rich positional semantic
dependencies between context and image while avoiding IR. Extensive
experimental results show the proposed HAAP achieves state-of-the-art (SOTA)
performance in terms of accuracy, complexity, and latency on several datasets.
