  Modern online platforms rely on effective rating systems to learn about
items. We consider the optimal design of rating systems that collect binary
feedback after transactions. We make three contributions. First, we formalize
the performance of a rating system as the speed with which it recovers the true
underlying ranking on items (in a large deviations sense), accounting for both
items' underlying match rates and the platform's preferences. Second, we
provide an efficient algorithm to compute the binary feedback system that
yields the highest such performance. Finally, we show how this theoretical
perspective can be used to empirically design an implementable, approximately
optimal rating system, and validate our approach using real-world experimental
data collected on Amazon Mechanical Turk.
