  Many state-of-the-art causal discovery methods aim to generate an output
graph that encodes the graphical separation and connection statements of the
causal graph that underlies the data-generating process. In this work, we argue
that an evaluation of a causal discovery method against synthetic data should
include an analysis of how well this explicit goal is achieved by measuring how
closely the separations/connections of the method's output align with those of
the ground truth. We show that established evaluation measures do not
accurately capture the difference in separations/connections of two causal
graphs, and we introduce three new measures of distance called s/c-distance,
Markov distance and Faithfulness distance that address this shortcoming. We
complement our theoretical analysis with toy examples, empirical experiments
and pseudocode.
