  The current graph neural network (GNN) systems have established a clear trend
of not showing training accuracy results, and directly or indirectly relying on
smaller datasets for evaluations majorly. Our in-depth analysis shows that it
leads to a chain of pitfalls in the system design and evaluation process,
questioning the practicality of many of the proposed system optimizations, and
affecting conclusions and lessons learned. We analyze many single-GPU systems
and show the fundamental impact of these pitfalls. We further develop
hypotheses, recommendations, and evaluation methodologies, and provide future
directions. Finally, a new reference system is developed to establish a new
line of optimizations rooted in solving the system-design pitfalls efficiently
and practically. The proposed design can productively be integrated into prior
works, thereby truly advancing the state-of-the-art.
