  Constrained Markov processes, such as reflecting diffusions, behave as an
unconstrained process in the interior of a domain but upon reaching the
boundary are controlled in some way so that they do not leave the closure of
the domain. In this paper, the behavior in the interior is specified by a
generator of a Markov process, and the constraints are specified by a
controlled generator. Together, the generators define a "constrained martingale
problem". The desired constrained processes are constructed by first solving a
simpler "controlled martingale problem" and then obtaining the desired process
as a time-change of the controlled process. As for ordinary martingale
problems, it is rarely obvious that the process constructed in this manner is
unique. The primary goal of the paper is to show that from among the processes
constructed in this way one can "select", in the sense of Krylov, a strong
Markov process. Corollaries to these constructions include the observation that
uniqueness among strong Markov solutions implies uniqueness among all
solutions. These results provide useful tools for proving uniqueness for
constrained processes including reflecting diffusions. The constructions also
yield viscosity semisolutions of the resolvent equation and, if uniqueness
holds, a viscosity solution, without proving a comparison principle. We
illustrate our results by applying them to reflecting diffusions in piecewise
smooth domains. We prove existence of a strong Markov solution to the SDE with
reflection, under conditions more general than in Dupuis and Ishii (1993): In
fact our conditions are known to be optimal in the case of simple, convex
polyhedrons with constant direction of reflection on each face (Dai and
Williams (1995)). We also indicate how the results can be applied to processes
with Wentzell boundary conditions and nonlocal boundary conditions.
