  With neural architecture search methods gaining ground on manually designed
deep neural networks -even more rapidly as model sophistication escalates-, the
research trend shifts towards arranging different and often increasingly
complex neural architecture search spaces. In this conjuncture, delineating
algorithms which can efficiently explore these search spaces can result in a
significant improvement over currently used methods, which, in general,
randomly select the structural variation operator, hoping for a performance
gain. In this paper, we investigate the effect of different variation operators
in a complex domain, that of multi-network heterogeneous neural models. These
models have an extensive and complex search space of structures as they require
multiple sub-networks within the general model in order to answer to different
output types. From that investigation, we extract a set of general guidelines,
whose application is not limited to that particular type of model, and are
useful to determine the direction in which an architecture optimization method
could find the largest improvement. To deduce the set of guidelines, we
characterize both the variation operators, according to their effect on the
complexity and performance of the model; and the models, relying on diverse
metrics which estimate the quality of the different parts composing it.
