  Training data memorization in language models impacts model capability
(generalization) and safety (privacy risk). This paper focuses on analyzing
prompts' impact on detecting the memorization of 6 masked language model-based
named entity recognition models. Specifically, we employ a diverse set of 400
automatically generated prompts, and a pairwise dataset where each pair
consists of one person's name from the training set and another name out of the
set. A prompt completed with a person's name serves as input for getting the
model's confidence in predicting this name. Finally, the prompt performance of
detecting model memorization is quantified by the percentage of name pairs for
which the model has higher confidence for the name from the training set. We
show that the performance of different prompts varies by as much as 16
percentage points on the same model, and prompt engineering further increases
the gap. Moreover, our experiments demonstrate that prompt performance is
model-dependent but does generalize across different name sets. A comprehensive
analysis indicates how prompt performance is influenced by prompt properties,
contained tokens, and the model's self-attention weights on the prompt.
