  $L_0$ regularization of neural networks is a fundamental problem. In addition
to regularizing models for better generalizability, $L_0$ regularization also
applies to selecting input features and training sparse neural networks. There
is a large body of research on related topics, some with quite complicated
methods. In this paper, we show that a straightforward formulation, BinMask,
which multiplies weights with deterministic binary masks and uses the identity
straight-through estimator for backpropagation, is an effective $L_0$
regularizer. We evaluate BinMask on three tasks: feature selection, network
sparsification, and model regularization. Despite its simplicity, BinMask
achieves competitive performance on all the benchmarks without task-specific
tuning compared to methods designed for each task. Our results suggest that
decoupling weights from mask optimization, which has been widely adopted by
previous work, is a key component for effective $L_0$ regularization.
