  We adapt the Douglas-Rachford (DR) splitting method to solve nonconvex
feasibility problems by studying this method for a class of nonconvex
optimization problem. While the convergence properties of the method for convex
problems have been well studied, far less is known in the nonconvex setting. In
this paper, for the direct adaptation of the method to minimize the sum of a
proper closed function $g$ and a smooth function $f$ with a Lipschitz
continuous gradient, we show that if the step-size parameter is smaller than a
computable threshold and the sequence generated has a cluster point, then it
gives a stationary point of the optimization problem. Convergence of the whole
sequence and a local convergence rate are also established under the additional
assumption that $f$ and $g$ are semi-algebraic. We also give simple sufficient
conditions guaranteeing the boundedness of the sequence generated. We then
apply our nonconvex DR splitting method to finding a point in the intersection
of a closed convex set $C$ and a general closed set $D$ by minimizing the
squared distance to $C$ subject to $D$. We show that if either set is bounded
and the step-size parameter is smaller than a computable threshold, then the
sequence generated from the DR splitting method is actually bounded.
Consequently, the sequence generated will have cluster points that are
stationary for an optimization problem, and the whole sequence is convergent
under an additional assumption that $C$ and $D$ are semi-algebraic. We achieve
these results based on a new merit function constructed particularly for the DR
splitting method. Our preliminary numerical results indicate that our DR
splitting method usually outperforms the alternating projection method in
finding a sparse solution of a linear system, in terms of both the solution
quality and the number of iterations taken.
