  Relation extraction (RE) aims to extract the relations between entity names
from the textual context. In principle, textual context determines the
ground-truth relation and the RE models should be able to correctly identify
the relations reflected by the textual context. However, existing work has
found that the RE models memorize the entity name patterns to make RE
predictions while ignoring the textual context. This motivates us to raise the
question: ``are RE models robust to the entity replacements?'' In this work, we
operate the random and type-constrained entity replacements over the RE
instances in TACRED and evaluate the state-of-the-art RE models under the
entity replacements. We observe the 30\% - 50\% F1 score drops on the
state-of-the-art RE models under entity replacements. These results suggest
that we need more efforts to develop effective RE models robust to entity
replacements. We release the source code at
https://github.com/wangywUST/RobustRE.
