  Companies survey their customers to measure their satisfaction levels with
the company and its services. The received responses are crucial as they allow
companies to assess their respective performances and find ways to make needed
improvements. This study focuses on the non-systematic bias that arises when
customers assign numerical values in ordinal surveys. Using real customer
satisfaction survey data of a large retail bank, we show that the common
practice of segmenting ordinal survey responses into uneven segments limit the
value that can be extracted from the data. We then show that it is possible to
assess the magnitude of the irreducible error under simple assumptions, even in
real surveys, and place the achievable modeling goal in perspective. We finish
the study by suggesting that a thoughtful survey design, which uses either a
careful binning strategy or proper calibration, can reduce the compounding
non-systematic error even in elaborated ordinal surveys. A possible application
of the calibration method we propose is efficiently conducting targeted surveys
using active learning.
