  This study investigates the behaviors of Large Language Models (LLMs) when
faced with conflicting prompts versus their internal memory. This will not only
help to understand LLMs' decision mechanism but also benefit real-world
applications, such as retrieval-augmented generation (RAG). Drawing on
cognitive theory, we target the first scenario of decision-making styles where
there is no superiority in the conflict and categorize LLMs' preference into
dependent, intuitive, and rational/irrational styles. Another scenario of
factual robustness considers the correctness of prompt and memory in
knowledge-intensive tasks, which can also distinguish if LLMs behave rationally
or irrationally in the first scenario. To quantify them, we establish a
complete benchmarking framework including a dataset, a robustness evaluation
pipeline, and corresponding metrics. Extensive experiments with seven LLMs
reveal their varying behaviors. And, with role play intervention, we can change
the styles, but different models present distinct adaptivity and upper-bound.
One of our key takeaways is to optimize models or the prompts according to the
identified style. For instance, RAG models with high role play adaptability may
dynamically adjust the interventions according to the quality of retrieval
results -- being dependent to better leverage informative context; and, being
intuitive when external prompt is noisy.
