  Recent work shows that causal facts can be effectively extracted from LLMs
through prompting, facilitating the creation of causal graphs for causal
inference tasks. However, it is unclear if this success is limited to
explicitly-mentioned causal facts in the pretraining data which the model can
memorize. Thus, this work investigates: Can LLMs infer causal relations from
other relational data in text? To disentangle the role of memorized causal
facts vs inferred causal relations, we finetune LLMs on synthetic data
containing temporal, spatial and counterfactual relations, and measure whether
the LLM can then infer causal relations. We find that: (a) LLMs are susceptible
to inferring causal relations from the order of two entity mentions in text
(e.g. X mentioned before Y implies X causes Y); (b) if the order is randomized,
LLMs still suffer from the post hoc fallacy, i.e. X occurs before Y (temporal
relation) implies X causes Y. We also find that while LLMs can correctly deduce
the absence of causal relations from temporal and spatial relations, they have
difficulty inferring causal relations from counterfactuals, questioning their
understanding of causality.
