  The prevailing issue of factual inconsistency errors in conventional
Retrieval Augmented Generation (RAG) motivates the study of Factual Consistency
Evaluation (FCE). Despite the various FCE methods proposed earlier, these
methods are evaluated on datasets generated by specific Large Language Models
(LLMs). Without a comprehensive benchmark, it remains unexplored how these FCE
methods perform on other LLMs with different error distributions or even unseen
error types, as these methods may fail to detect the error types generated by
other LLMs. To fill this gap, in this paper, we propose the first comprehensive
FCE benchmark \emph{Face4RAG} for RAG independent of the underlying LLM. Our
benchmark consists of a synthetic dataset built upon a carefully designed
typology for factuality inconsistency error and a real-world dataset
constructed from six commonly used LLMs, enabling evaluation of FCE methods on
specific error types or real-world error distributions. On the proposed
benchmark, we discover the failure of existing FCE methods to detect the
logical fallacy, which refers to a mismatch of logic structures between the
answer and the retrieved reference. To fix this issue, we further propose a new
method called \emph{L-Face4RAG} with two novel designs of logic-preserving
answer decomposition and fact-logic FCE. Extensive experiments show L-Face4RAG
substantially outperforms previous methods for factual inconsistency detection
on a wide range of tasks, notably beyond the RAG task from which it is
originally motivated. Both the benchmark and our proposed method are publicly
available.\footnote{\url{https://huggingface.co/datasets/yq27/Face4RAG}\label{link_face4rag}}
