  We consider optimal sensor scheduling with unknown communication channel
statistics. We formulate two types of scheduling problems with the
communication rate being a soft or hard constraint, respectively. We first
present some structural results on the optimal scheduling policy using dynamic
programming and assuming the channel statistics is known. We prove that the
Q-factor is monotonic and submodular, which leads to the threshold-like
structures in both types of problems. Then we develop a stochastic
approximation and parameter learning frameworks to deal with the two scheduling
problems with unknown channel statistics. We utilize their structures to design
specialized learning algorithms. We prove the convergence of these algorithms.
Performance improvement compared with the standard Q-learning algorithm is
shown through numerical examples.
