  We consider the problem of source sampling and transmission scheduling for
age-of-information minimization in a system consisting of multiple energy
harvesting (EH) sources and a sink node. At each time, one of the sources is
selected by the scheduler and the quality of its channel to the sink is
measured. This probed channel quality is then used to decide whether a source
will sample an observation and transmit the packet to the sink in that time
slot. We formulate this problem as a constrained Markov decision process (CMDP)
assuming i.i.d. energy arrival and channel fading processes, and relax it using
a Lagrange multiplier. We apply a near optimal Whittle's index policy to decide
the node to be probed. Next, for the probed node, we derive an optimal
threshold policy, which recommends source sampling and observation transmission
from the probed source only when the measured channel quality is above a
threshold. Our proposed policy is called Whittle's index and threshold based
source scheduling and sampling (WITS3) policy. However, in order to calculate
Whittle's indices, one must be aware of the underlying processes' transition
matrices, which are occasionally concealed from the scheduler. Therefore, we
further propose a variant Q-WITS3 of WITS3 based on Q-learning assisted by two
timescale asynchronous stochastic approximation, which seeks to learn Whittle's
indices and optimal policies for the case with unknown channel states and EH
characteristics. Numerical results demonstrate the efficacy of our algorithms
over two baseline policies.
