  The two most commonly used criteria for assessing causal model discovery with
artificial data are edit-distance and Kullback-Leibler divergence, measured
from the true model to the learned model. Both of these metrics maximally
reward the true model. However, we argue that they are both insufficiently
discriminating in judging the relative merits of false models. Edit distance,
for example, fails to distinguish between strong and weak probabilistic
dependencies. KL divergence, on the other hand, rewards equally all
statistically equivalent models, regardless of their different causal claims.
We propose an augmented KL divergence, which we call Causal KL (CKL), which
takes into account causal relationships which distinguish between
observationally equivalent models. Results are presented for three variants of
CKL, showing that Causal KL works well in practice.
