  "The chance to win given a certain move" is an easily obtainable quantity
from data and often quoted in gaming statistics. It is also the fundamental
quantity that reinforcement learning AI bases on. Unfortunately, this
conditional probability can be misleading. Unless all players are equally
skilled, this number does not tell us the intrinsic value of such move. That is
because conditioning on one good move also inevitably selects a subset of
better players. They tend to make other good moves, which also contribute to
the extra winning chance. We present a simple toy model to quantify this "skill
bias" effect, and then propose a general method to remove it. Our method is
modular, generalizable, and also only requires easily obtainable quantities
from data. In particular, it gets the same answer independent of whether the
data comes from a group of good or bad players. This may help us to eventually
break free from the conventional wisdom of "learning from the experts" and
avoid the Group Thinking pitfall.
