  Recent research has observed that in machine learning optimization, gradient
descent (GD) often operates at the edge of stability (EoS) [Cohen, et al.,
2021], where the stepsizes are set to be large, resulting in non-monotonic
losses induced by the GD iterates. This paper studies the convergence and
implicit bias of constant-stepsize GD for logistic regression on linearly
separable data in the EoS regime. Despite the presence of local oscillations,
we prove that the logistic loss can be minimized by GD with \emph{any} constant
stepsize over a long time scale. Furthermore, we prove that with \emph{any}
constant stepsize, the GD iterates tend to infinity when projected to a
max-margin direction (the hard-margin SVM direction) and converge to a fixed
vector that minimizes a strongly convex potential when projected to the
orthogonal complement of the max-margin direction. In contrast, we also show
that in the EoS regime, GD iterates may diverge catastrophically under the
exponential loss, highlighting the superiority of the logistic loss. These
theoretical findings are in line with numerical simulations and complement
existing theories on the convergence and implicit bias of GD for logistic
regression, which are only applicable when the stepsizes are sufficiently
small.
