  Since real-world objects and their interactions are often multi-modal and
multi-typed, heterogeneous networks have been widely used as a more powerful,
realistic, and generic superclass of traditional homogeneous networks (graphs).
Meanwhile, representation learning (\aka~embedding) has recently been
intensively studied and shown effective for various network mining and
analytical tasks. In this work, we aim to provide a unified framework to deeply
summarize and evaluate existing research on heterogeneous network embedding
(HNE), which includes but goes beyond a normal survey. Since there has already
been a broad body of HNE algorithms, as the first contribution of this work, we
provide a generic paradigm for the systematic categorization and analysis over
the merits of various existing HNE algorithms. Moreover, existing HNE
algorithms, though mostly claimed generic, are often evaluated on different
datasets. Understandable due to the application favor of HNE, such indirect
comparisons largely hinder the proper attribution of improved task performance
towards effective data preprocessing and novel technical design, especially
considering the various ways possible to construct a heterogeneous network from
real-world application data. Therefore, as the second contribution, we create
four benchmark datasets with various properties regarding scale, structure,
attribute/label availability, and \etc.~from different sources, towards handy
and fair evaluations of HNE algorithms. As the third contribution, we carefully
refactor and amend the implementations and create friendly interfaces for 13
popular HNE algorithms, and provide all-around comparisons among them over
multiple tasks and experimental settings.
