  Semantic textual similarity is one of the open research challenges in the
field of Natural Language Processing. Extensive research has been carried out
in this field and near-perfect results are achieved by recent transformer-based
models in existing benchmark datasets like the STS dataset and the SICK
dataset. In this paper, we study the sentences in these datasets and analyze
the sensitivity of various word embeddings with respect to the complexity of
the sentences. We build a complex sentences dataset comprising of 50 sentence
pairs with associated semantic similarity values provided by 15 human
annotators. Readability analysis is performed to highlight the increase in
complexity of the sentences in the existing benchmark datasets and those in the
proposed dataset. Further, we perform a comparative analysis of the performance
of various word embeddings and language models on the existing benchmark
datasets and the proposed dataset. The results show the increase in complexity
of the sentences has a significant impact on the performance of the embedding
models resulting in a 10-20% decrease in Pearson's and Spearman's correlation.
