  Modern predictive models are often deployed to environments in which
computational budgets are dynamic. Anytime algorithms are well-suited to such
environments as, at any point during computation, they can output a prediction
whose quality is a function of computation time. Early-exit neural networks
have garnered attention in the context of anytime computation due to their
capability to provide intermediate predictions at various stages throughout the
network. However, we demonstrate that current early-exit networks are not
directly applicable to anytime settings, as the quality of predictions for
individual data points is not guaranteed to improve with longer computation. To
address this shortcoming, we propose an elegant post-hoc modification, based on
the Product-of-Experts, that encourages an early-exit network to become
gradually confident. This gives our deep models the property of conditional
monotonicity in the prediction quality -- an essential stepping stone towards
truly anytime predictive modeling using early-exit architectures. Our empirical
results on standard image-classification tasks demonstrate that such behaviors
can be achieved while preserving competitive accuracy on average.
