  In his "ship of state" allegory (\textit{Republic}, Book VI, 488) Plato poses
a question -- how can a crew of sailors presumed to know little about the art
of navigation recognize the true pilot among them? The allegory argues that a
simple majority voting procedure cannot safely determine who is most qualified
to pilot a ship when the voting members are ignorant or biased. We formalize
Plato's concerns by considering the problem in AI safety of monitoring noisy AI
agents in unsupervised settings. An algorithm evaluating AI agents using
unlabeled data would be subject to the evaluation dilemma - how would we know
the evaluation algorithm was correct itself? This endless validation chain can
be avoided by considering purely algebraic functions of the observed responses.
We can construct complete postulates than can prove or disprove the logical
consistency of any grading algorithm. A complete set of postulates exists
whenever we are evaluating $N$ experts that took $T$ tests with $Q$ questions
with $R$ responses each. We discuss evaluating binary classifiers that have
taken a single test - the $(N,T=1,Q,R=2)$ tests. We show how some of the
postulates have been previously identified in the ML literature but not
recognized as such - the \textbf{agreement equations} of Platanios. The
complete postulates for pair correlated binary classifiers are considered and
we show how it allows for error correlations to be quickly calculated. An
algebraic evaluator based on the assumption that the ensemble is error
independent is compared with grading by majority voting on evaluations using
the \uciadult and and \texttt{two-norm} datasets. Throughout, we demonstrate
how the formalism of logical consistency via algebraic postulates of evaluation
can help increase the safety of machines using AI algorithms.
