  In this paper, we study possible extensions of the main ideas and methods of
constrained DC optimization to the case of nonlinear semidefinite programming
problems and more general nonlinear and nonsmooth cone constrained optimization
problems.
  In the first part of the paper, we analyse two different approaches to the
definition of DC matrix-valued functions (namely, order-theoretic and
componentwise), study some properties of convex and DC matrix-valued mappings
and demonstrate how to compute DC decompositions of some nonlinear semidefinite
constraints appearing in applications. We also compute a DC decomposition of
the maximal eigenvalue of a DC matrix-valued function. This DC decomposition
can be used to reformulate DC semidefinite constraints as DC inequality
constrains. Finally, we study local optimality conditions for general cone
constrained DC optimization problems.
  The second part of the paper is devoted to a detailed convergence analysis of
two extensions of the well-known DCA method for solving DC (Difference of
Convex functions) optimization problems to the case of general cone constrained
DC optimization problems. We study the global convergence of the DCA for cone
constrained problems and present a comprehensive analysis of a version of the
DCA utilizing exact penalty functions. In particular, we study the exactness
property of the penalized convex subproblems and provide two types of
sufficient conditions for the convergence of the exact penalty method to a
feasible and critical point of a cone constrained DC optimization problem from
an infeasible starting point. In the numerical section of this work, the exact
penalty DCA is applied to the problem of computing compressed modes for
variational problems and the sphere packing problem on Grassmannian.
