  Regularization is a critical component in deep learning training, with weight
decay being a commonly used approach. It applies a constant penalty coefficient
uniformly across all parameters. This may be unnecessarily restrictive for some
parameters, while insufficiently restricting others. To dynamically adjust
penalty coefficients for different parameter groups, we present constrained
parameter regularization (CPR) as an alternative to traditional weight decay.
Instead of applying a single constant penalty to all parameters, we enforce an
upper bound on a statistical measure (e.g., the L$_2$-norm) of parameter
groups. Consequently, learning becomes a constraint optimization problem, which
we address by an adaptation of the augmented Lagrangian method. CPR only
requires two hyperparameters and incurs no measurable runtime overhead.
Additionally, we propose a simple but efficient mechanism to adapt the upper
bounds during the optimization. We provide empirical evidence of CPR's efficacy
in experiments on the "grokking" phenomenon, computer vision, and language
modeling tasks. Our results demonstrate that CPR counteracts the effects of
grokking and consistently matches or outperforms traditional weight decay.
