  Rewards and punishments in different forms are pervasive and present in a
wide variety of decision-making scenarios. By observing the outcome of a
sufficient number of repeated trials, one would gradually learn the value and
usefulness of a particular policy or strategy. However, in a given environment,
the outcomes resulting from different trials are subject to chance influence
and variations. In learning about the usefulness of a given policy, significant
costs are involved in systematically undertaking the sequential trials;
therefore, in most learning episodes, one would wish to keep the cost within
bounds by adopting learning stopping rules. In this paper, we examine the
deployment of different stopping strategies in given learning environments
which vary from highly stringent for mission critical operations to highly
tolerant for non-mission critical operations, and emphasis is placed on the
former with particular application to aviation safety. In policy evaluation,
two sequential phases of learning are identified, and we describe the outcomes
variations using a probabilistic model, with closedform expressions obtained
for the key measures of performance. Decision rules that map the trial
observations to policy choices are also formulated. In addition, simulation
experiments are performed, which corroborate the validity of the theoretical
results.
