  Scientific papers are complex and understanding the usefulness of these
papers requires prior knowledge. Peer reviews are comments on a paper provided
by designated experts on that field and hold a substantial amount of
information, not only for the editors and chairs to make the final decision,
but also to judge the potential impact of the paper. In this paper, we propose
to use aspect-based sentiment analysis of scientific reviews to be able to
extract useful information, which correlates well with the accept/reject
decision.
  While working on a dataset of close to 8k reviews from ICLR, one of the top
conferences in the field of machine learning, we use an active learning
framework to build a training dataset for aspect prediction, which is further
used to obtain the aspects and sentiments for the entire dataset. We show that
the distribution of aspect-based sentiments obtained from a review is
significantly different for accepted and rejected papers. We use the aspect
sentiments from these reviews to make an intriguing observation, certain
aspects present in a paper and discussed in the review strongly determine the
final recommendation. As a second objective, we quantify the extent of
disagreement among the reviewers refereeing a paper. We also investigate the
extent of disagreement between the reviewers and the chair and find that the
inter-reviewer disagreement may have a link to the disagreement with the chair.
One of the most interesting observations from this study is that reviews, where
the reviewer score and the aspect sentiments extracted from the review text
written by the reviewer are consistent, are also more likely to be concurrent
with the chair's decision.
