  Rich entity representations are useful for a wide class of problems involving
entities. Despite their importance, there is no standardized benchmark that
evaluates the overall quality of entity representations. In this work, we
propose EntEval: a test suite of diverse tasks that require nontrivial
understanding of entities including entity typing, entity similarity, entity
relation prediction, and entity disambiguation. In addition, we develop
training techniques for learning better entity representations by using natural
hyperlink annotations in Wikipedia. We identify effective objectives for
incorporating the contextual information in hyperlinks into state-of-the-art
pretrained language models and show that they improve strong baselines on
multiple EntEval tasks.
