  Probing Pre-trained Language Models (PLMs) using prompts has indirectly
implied that language models (LMs) can be treated as knowledge bases. To this
end, this phenomena has been effective especially when these LMs are fine-tuned
towards not just data of a specific domain, but also to the style or linguistic
pattern of the prompts themselves. We observe that, satisfying a particular
linguistic pattern in prompts is an unsustainable constraint that unnecessarily
lengthens the probing task, especially because, they are often manually
designed and the range of possible prompt template patterns can vary depending
on the prompting objective and domain. We therefore explore an idea of using a
position-attention mechanism to capture positional information of each word in
a prompt relative to the mask to be filled, hence avoiding the need to
re-construct prompts when the prompts linguistic pattern changes. Using our
approach, we demonstrate the ability of eliciting answers to rare prompt
templates (in a case study on health outcome generation) such as Postfix and
Mixed patterns whose missing information is respectively at the start and in
multiple random places of the prompt. More so, using various biomedical PLMs,
our approach consistently outperforms a baseline in which the default mask
language model (MLM) representation is used to predict masked tokens.
