  Neural network-based embeddings have been the mainstream approach for
creating a vector representation of the text to capture lexical and semantic
similarities and dissimilarities. In general, existing encoding methods dismiss
the punctuation as insignificant information; consequently, they are routinely
treated as a predefined token/word or eliminated in the pre-processing phase.
However, punctuation could play a significant role in the semantics of the
sentences, as in "Let's eat\hl{,} grandma" and "Let's eat grandma". We
hypothesize that a punctuation-aware representation model would affect the
performance of the downstream tasks. Thereby, we propose a model-agnostic
method that incorporates both syntactic and contextual information to improve
the performance of the sentiment classification task. We corroborate our
findings by conducting experiments on publicly available datasets and provide
case studies that our model generates representations with respect to the
punctuation in the sentence.
