  To handle graphs in which features or connectivities are evolving over time,
a series of temporal graph neural networks (TGNNs) have been proposed. Despite
the success of these TGNNs, the previous TGNN evaluations reveal several
limitations regarding four critical issues: 1) inconsistent datasets, 2)
inconsistent evaluation pipelines, 3) lacking workload diversity, and 4)
lacking efficient comparison. Overall, there lacks an empirical study that puts
TGNN models onto the same ground and compares them comprehensively. To this
end, we propose BenchTemp, a general benchmark for evaluating TGNN models on
various workloads. BenchTemp provides a set of benchmark datasets so that
different TGNN models can be fairly compared. Further, BenchTemp engineers a
standard pipeline that unifies the TGNN evaluation. With BenchTemp, we
extensively compare the representative TGNN models on different tasks (e.g.,
link prediction and node classification) and settings (transductive and
inductive), w.r.t. both effectiveness and efficiency metrics. We have made
BenchTemp publicly available at https://github.com/qianghuangwhu/benchtemp.
