  Large models and enormous data are essential driving forces of the
unprecedented successes achieved by modern algorithms, especially in scientific
computing and machine learning. Nevertheless, the growing dimensionality and
model complexity, as well as the non-negligible workload of data
pre-processing, also bring formidable costs to such successes in both
computation and data aggregation. As the deceleration of Moore's Law slackens
the cost reduction of computation from the hardware level, fast heuristics for
expensive classical routines and efficient algorithms for exploiting limited
data are increasingly indispensable for pushing the limit of algorithm potency.
This thesis explores some of such algorithms for fast execution and efficient
data utilization.
  From the computational efficiency perspective, we design and analyze fast
randomized low-rank decomposition algorithms for large matrices based on
"matrix sketching", which can be regarded as a dimension reduction strategy in
the data space. These include the randomized pivoting-based interpolative and
CUR decomposition discussed in Chapter 2 and the randomized subspace
approximations discussed in Chapter 3.
  From the sample efficiency perspective, we focus on learning algorithms with
various incorporations of data augmentation that improve generalization and
distributional robustness provably. Specifically, Chapter 4 presents a sample
complexity analysis for data augmentation consistency regularization where we
view sample efficiency from the lens of dimension reduction in the function
space. Then in Chapter 5, we introduce an adaptively weighted data augmentation
consistency regularization algorithm for distributionally robust optimization
with applications in medical image segmentation.
