  Language models are increasingly becoming popular in AI-powered scientific IR
systems. This paper evaluates popular scientific language models in handling
(i) short-query texts and (ii) textual neighbors. Our experiments showcase the
inability to retrieve relevant documents for a short-query text even under the
most relaxed conditions. Additionally, we leverage textual neighbors, generated
by small perturbations to the original text, to demonstrate that not all
perturbations lead to close neighbors in the embedding space. Further, an
exhaustive categorization yields several classes of orthographically and
semantically related, partially related, and completely unrelated neighbors.
Retrieval performance turns out to be more influenced by the surface form
rather than the semantics of the text.
