  Sparsity-constraint optimization has wide applicability in signal processing,
statistics, and machine learning. Existing fast algorithms must burdensomely
tune parameters, such as the step size or the implementation of precise stop
criteria, which may be challenging to determine in practice. To address this
issue, we develop an algorithm named Sparsity-Constraint Optimization via
sPlicing itEration (SCOPE) to optimize nonlinear differential objective
functions with strong convexity and smoothness in low dimensional subspaces.
Algorithmically, the SCOPE algorithm converges effectively without tuning
parameters. Theoretically, SCOPE has a linear convergence rate and converges to
a solution that recovers the true support set when it correctly specifies the
sparsity. We also develop parallel theoretical results without
restricted-isometry-property-type conditions. We apply SCOPE's versatility and
power to solve sparse quadratic optimization, learn sparse classifiers, and
recover sparse Markov networks for binary variables. The numerical results on
these specific tasks reveal that SCOPE perfectly identifies the true support
set with a 10--1000 speedup over the standard exact solver, confirming SCOPE's
algorithmic and theoretical merits. Our open-source Python package skscope
based on C++ implementation is publicly available on GitHub, reaching a
ten-fold speedup on the competing convex relaxation methods implemented by the
cvxpy library.
