  Adversarial training is an important topic in robust deep learning, but the
community lacks attention to its practical usage. In this paper, we aim to
resolve a real-world challenge, i.e., training a model on an imbalanced and
noisy dataset to achieve high clean accuracy and adversarial robustness, with
our proposed Omnipotent Adversarial Training (OAT) strategy. OAT consists of
two innovative methodologies to address the imperfection in the training set.
We first introduce an oracle into the adversarial training process to help the
model learn a correct data-label conditional distribution. This
carefully-designed oracle can provide correct label annotations for adversarial
training. We further propose logits adjustment adversarial training to overcome
the data imbalance issue, which can help the model learn a Bayes-optimal
distribution. Our comprehensive evaluation results show that OAT outperforms
other baselines by more than 20% clean accuracy improvement and 10% robust
accuracy improvement under complex combinations of data imbalance and label
noise scenarios. The code can be found in https://github.com/GuanlinLee/OAT.
