  Linear classifiers and leaky ReLU networks trained by gradient flow on the
logistic loss have an implicit bias towards solutions which satisfy the
Karush--Kuhn--Tucker (KKT) conditions for margin maximization. In this work we
establish a number of settings where the satisfaction of these KKT conditions
implies benign overfitting in linear classifiers and in two-layer leaky ReLU
networks: the estimators interpolate noisy training data and simultaneously
generalize well to test data. The settings include variants of the noisy
class-conditional Gaussians considered in previous work as well as new
distributional settings where benign overfitting has not been previously
observed. The key ingredient to our proof is the observation that when the
training data is nearly-orthogonal, both linear classifiers and leaky ReLU
networks satisfying the KKT conditions for their respective margin maximization
problems behave like a nearly uniform average of the training examples.
