  We note that decoupled weight decay regularization is a particular case of
weight norm control where the target norm of weights is set to 0. Any
optimization method (e.g., Adam) which uses decoupled weight decay
regularization (respectively, AdamW) can be viewed as a particular case of a
more general algorithm with weight norm control (respectively, AdamWN). We
argue that setting the target norm of weights to 0 can be suboptimal and other
target norm values can be considered. For instance, any training run where
AdamW achieves a particular norm of weights can be challenged by AdamWN
scheduled to achieve a comparable norm of weights. We discuss various
implications of introducing weight norm control instead of weight decay.
