  Optimal control theory is developed for the task of obtaining a primary
objective in a subspace of the Hilbert space while avoiding other subspaces of
the Hilbert space. The primary objective can be a state-to-state transition or
a unitary transformation. A new optimization functional is introduced which
leads to monotonic convergence of the algorithm. This approach becomes
necessary for molecular systems subject to processes implying loss of coherence
such as predissociation or ionization. In these subspaces controllability is
hampered or even completely lost. Avoiding the lossy channels is achieved via a
functional constraint which depends on the state of the system at each instant
in time. We outline the resulting new algorithm, discuss its convergence
properties and demonstrate its functionality for the example of a
state-to-state transition and of a unitary transformation for a model of cold
Rb2.
