  The scaling law, a strategy that involves the brute-force scaling of the
training dataset and learnable parameters, has become a prevalent approach for
developing stronger learning models. In this paper, we examine its rationale in
terms of learning from relational graphs. We demonstrate that directly adhering
to such a scaling law does not necessarily yield stronger models due to
architectural incompatibility and representation bottlenecks. To tackle this
challenge, we propose a novel framework for learning from relational graphs via
knowledge-aware parsimony learning. Our method draws inspiration from the
duality between data and knowledge inherent in these graphs. Specifically, we
first extract knowledge (like symbolic logic and physical laws) during the
learning process, and then apply combinatorial generalization to the task at
hand. This extracted knowledge serves as the ``building blocks'' for achieving
parsimony learning. By applying this philosophy to architecture, parameters,
and inference, we can effectively achieve versatile, sample-efficient, and
interpretable learning. Experimental results show that our proposed framework
surpasses methods that strictly follow the traditional scaling-up roadmap. This
highlights the importance of incorporating knowledge in the development of
next-generation learning technologies.
