  Recent advances in Deep Learning have greatly improved performance on various
tasks such as object detection, image segmentation, sentiment analysis. The
focus of most research directions up until very recently has been on beating
state-of-the-art results. This has materialized in the utilization of bigger
and bigger models and techniques which help the training procedure to extract
more predictive power out of a given dataset. While this has lead to great
results, many of which with real-world applications, other relevant aspects of
deep learning have remained neglected and unknown. In this work, we will
present several novel deep learning training techniques which, while capable of
offering significant performance gains they also reveal several interesting
analysis results regarding convergence speed, optimization landscape
smoothness, and adversarial robustness. The methods presented in this work are
the following:
  $\bullet$ Perfect Ordering Approximation; a generalized model agnostic
curriculum learning approach. The results show the effectiveness of the
technique for improving training time as well as offer some new insight into
the training process of deep networks.
  $\bullet$ Cascading Sum Augmentation; an extension of mixup capable of
utilizing more data points for linear interpolation by leveraging a smoother
optimization landscape. This can be used for computer vision tasks in order to
improve both prediction performance as well as improve passive model
robustness.
