  Automated machine learning (AutoML) usually involves several crucial
components, such as Data Augmentation (DA) policy, Hyper-Parameter Optimization
(HPO), and Neural Architecture Search (NAS). Although many strategies have been
developed for automating these components in separation, joint optimization of
these components remains challenging due to the largely increased search
dimension and the variant input types of each component. In parallel to this,
the common practice of searching for the optimal architecture first and then
retraining it before deployment in NAS often suffers from low performance
correlation between the searching and retraining stages. An end-to-end solution
that integrates the AutoML components and returns a ready-to-use model at the
end of the search is desirable. In view of these, we propose DHA, which
achieves joint optimization of Data augmentation policy, Hyper-parameter and
Architecture. Specifically, end-to-end NAS is achieved in a differentiable
manner by optimizing a compressed lower-dimensional feature space, while DA
policy and HPO are regarded as dynamic schedulers, which adapt themselves to
the update of network parameters and network architecture at the same time.
Experiments show that DHA achieves state-of-the-art (SOTA) results on various
datasets and search spaces. To the best of our knowledge, we are the first to
efficiently and jointly optimize DA policy, NAS, and HPO in an end-to-end
manner without retraining.
