  We initiate the study of the truthfulness of calibration measures in
sequential prediction. A calibration measure is said to be truthful if the
forecaster (approximately) minimizes the expected penalty by predicting the
conditional expectation of the next outcome, given the prior distribution of
outcomes. Truthfulness is an important property of calibration measures,
ensuring that the forecaster is not incentivized to exploit the system with
deliberate poor forecasts. This makes it an essential desideratum for
calibration measures, alongside typical requirements, such as soundness and
completeness.
  We conduct a taxonomy of existing calibration measures and their
truthfulness. Perhaps surprisingly, we find that all of them are far from being
truthful. That is, under existing calibration measures, there are simple
distributions on which a polylogarithmic (or even zero) penalty is achievable,
while truthful prediction leads to a polynomial penalty. Our main contribution
is the introduction of a new calibration measure termed the Subsampled Smooth
Calibration Error (SSCE) under which truthful prediction is optimal up to a
constant multiplicative factor.
