  In natural language processing (NLP) we always rely on human judgement as the
golden quality evaluation method. However, there has been an ongoing debate on
how to better evaluate inter-rater reliability (IRR) levels for certain
evaluation tasks, such as translation quality evaluation (TQE), especially when
the data samples (observations) are very scarce. In this work, we first
introduce the study on how to estimate the confidence interval for the
measurement value when only one data (evaluation) point is available. Then,
this leads to our example with two human-generated observational scores, for
which, we introduce ``Student's \textit{t}-Distribution'' method and explain
how to use it to measure the IRR score using only these two data points, as
well as the confidence intervals (CIs) of the quality evaluation. We give
quantitative analysis on how the evaluation confidence can be greatly improved
by introducing more observations, even if only one extra observation. We
encourage researchers to report their IRR scores in all possible means, e.g.
using Student's \textit{t}-Distribution method whenever possible; thus making
the NLP evaluation more meaningful, transparent, and trustworthy. This
\textit{t}-Distribution method can be also used outside of NLP fields to
measure IRR level for trustworthy evaluation of experimental investigations,
whenever the observational data is scarce.
  Keywords: Inter-Rater Reliability (IRR); Scarce Observations; Confidence
Intervals (CIs); Natural Language Processing (NLP); Translation Quality
Evaluation (TQE); Student's \textit{t}-Distribution
