  In this paper, we study a continuous-time discounted jump Markov decision
process with both controlled actions and observations. The observation is only
available for a discrete set of time instances. At each time of observation,
one has to select an optimal timing for the next observation and a control
trajectory for the time interval between two observation points. We provide a
theoretical framework that the decision maker can utilize to find the optimal
observation epochs and the optimal actions jointly. Two cases are investigated.
One is gated queueing systems in which we explicitly characterize the optimal
action and the optimal observation where the optimal observation is shown to be
independent of the state. Another is the inventory control problem with Poisson
arrival process in which we obtain numerically the optimal action and
observation. The results show that it is optimal to observe more frequently at
a region of states where the optimal action adapts constantly.
