  Automated machine learning (AutoML) methods improve upon existing models by
optimizing various aspects of their design. While present methods focus on
hyperparameters and neural network topologies, other aspects of neural network
design can be optimized as well. To further the state of the art in AutoML,
this dissertation introduces techniques for discovering more powerful
activation functions and establishing more robust weight initialization for
neural networks. These contributions improve performance, but also provide new
perspectives on neural network optimization. First, the dissertation
demonstrates that discovering solutions specialized to specific architectures
and tasks gives better performance than reusing general approaches. Second, it
shows that jointly optimizing different components of neural networks is
synergistic, and results in better performance than optimizing individual
components alone. Third, it demonstrates that learned representations are
easier to optimize than hard-coded ones, creating further opportunities for
AutoML. The dissertation thus makes concrete progress towards fully automatic
machine learning in the future.
