  The graph of a Bayesian Network (BN) can be machine learned, determined by
causal knowledge, or a combination of both. In disciplines like bioinformatics,
applying BN structure learning algorithms can reveal new insights that would
otherwise remain unknown. However, these algorithms are less effective when the
input data are limited in terms of sample size, which is often the case when
working with real data. This paper focuses on purely machine learned and purely
knowledge-based BNs and investigates their differences in terms of graphical
structure and how well the implied statistical models explain the data. The
tests are based on four previous case studies whose BN structure was determined
by domain knowledge. Using various metrics, we compare the knowledge-based
graphs to the machine learned graphs generated from various algorithms
implemented in TETRAD spanning all three classes of learning. The results show
that, while the algorithms produce graphs with much higher model selection
score, the knowledge-based graphs are more accurate predictors of variables of
interest. Maximising score fitting is ineffective in the presence of limited
sample size because the fitting becomes increasingly distorted with limited
data, guiding algorithms towards graphical patterns that share higher fitting
scores and yet deviate considerably from the true graph. This highlights the
value of causal knowledge in these cases, as well as the need for more
appropriate fitting scores suitable for limited data. Lastly, the experiments
also provide new evidence that support the notion that results from simulated
data tell us little about actual real-world performance.
