  Most fact checking models for automatic fake news detection are based on
reasoning: given a claim with associated evidence, the models aim to estimate
the claim veracity based on the supporting or refuting content within the
evidence. When these models perform well, it is generally assumed to be due to
the models having learned to reason over the evidence with regards to the
claim. In this paper, we investigate this assumption of reasoning, by exploring
the relationship and importance of both claim and evidence. Surprisingly, we
find on political fact checking datasets that most often the highest
effectiveness is obtained by utilizing only the evidence, as the impact of
including the claim is either negligible or harmful to the effectiveness. This
highlights an important problem in what constitutes evidence in existing
approaches for automatic fake news detection.
