  We study self-similarity in random binary rooted trees. In a well-understood
case of Galton-Watson trees, a distribution on a space of trees is said to be
self-similar if it is invariant with respect to the operation of pruning, which
cuts the tree leaves. This only happens for the critical Galton-Watson tree (a
constant process progeny), which also exhibits other special symmetries. We
extend the prune-invariance setup to arbitrary binary trees with edge lengths.
In this general case the class of self-similar processes becomes much richer
and covers a variety of practically important situations. The main result is
construction of the hierarchical branching processes that satisfy various
self-similarity definitions (including mean self-similarity and self-similarity
in edge-lengths) depending on the process parameters. Taking the limit of
averaged stochastic dynamics, as the number of trajectories increases, we
obtain a deterministic system of differential equations that describes the
process evolution. This system is used to establish a phase transition that
separates fading and explosive behavior of the average process progeny. We
describe a class of critical Tokunaga processes that happen at the phase
transition boundary. They enjoy multiple additional symmetries and include the
celebrated critical binary Galton-Watson tree with independent exponential edge
length as a special case. Finally, we discuss a duality between trees and
continuous functions, and introduce a class of extreme-invariant processes,
constructed as the Harris paths of a self-similar hierarchical branching
process, whose local minima has the same (linearly scaled) distribution as the
original process.
