  Evaluating agent performance when outcomes are stochastic and agents use
randomized strategies can be challenging when there is limited data available.
The variance of sampled outcomes may make the simple approach of Monte Carlo
sampling inadequate. This is the case for agents playing heads-up no-limit
Texas hold'em poker, where man-machine competitions have involved multiple days
of consistent play and still not resulted in statistically significant
conclusions even when the winner's margin is substantial. In this paper, we
introduce AIVAT, a low variance, provably unbiased value assessment tool that
uses an arbitrary heuristic estimate of state value, as well as the explicit
strategy of a subset of the agents. Unlike existing techniques which reduce the
variance from chance events, or only consider game ending actions, AIVAT
reduces the variance both from choices by nature and by players with a known
strategy. The resulting estimator in no-limit poker can reduce the number of
hands needed to draw statistical conclusions by more than a factor of 10.
