  We study the evolution of artificial learning systems by means of selection.
Genetic programming is used to generate a sequence of populations of algorithms
which can be used by neural networks for supervised learning of a rule that
generates examples. In opposition to concentrating on final results, which
would be the natural aim while designing good learning algorithms, we study the
evolution process and pay particular attention to the temporal order of
appearance of functional structures responsible for the improvements in the
learning process, as measured by the generalization capabilities of the
resulting algorithms. The effect of such appearances can be described as
dynamical phase transitions. The concepts of phenotypic and genotypic
entropies, which serve to describe the distribution of fitness in the
population and the distribution of symbols respectively, are used to monitor
the dynamics. In different runs the phase transitions might be present or not,
with the system finding out good solutions, or staying in poor regions of
algorithm space. Whenever phase transitions occur, the sequence of appearances
are the same. We identify combinations of variables and operators which are
useful in measuring experience or performance in rule extraction and can thus
implement useful annealing of the learning schedule.
