  In this paper, we study a real-time monitoring system in which multiple
source nodes are responsible for sending update packets to a common destination
node in order to maintain the freshness of information at the destination.
Since it may not always be feasible to replace or recharge batteries in all
source nodes, we consider that the nodes are powered through wireless energy
transfer (WET) by the destination. For this system setup, we investigate the
optimal online sampling policy (referred to as the age-optimal policy) that
jointly optimizes WET and scheduling of update packet transmissions with the
objective of minimizing the long-term average weighted sum of
Age-of-Information (AoI) values for different physical processes (observed by
the source nodes) at the destination node, referred to as the sum-AoI. To solve
this optimization problem, we first model this setup as an average cost Markov
decision process (MDP). Due to the extreme curse of dimensionality in the state
space of the formulated MDP, classical reinforcement learning algorithms are no
longer applicable to our problem. Motivated by this, we propose a deep
reinforcement learning (DRL) algorithm that can learn the age-optimal policy in
a computationally-efficient manner. We further characterize the structural
properties of the age-optimal policy analytically, and demonstrate that it has
a threshold-based structure with respect to the AoI values for different
processes. We extend our analysis to characterize the structural properties of
the policy that maximizes average throughput for our system setup, referred to
as the throughput-optimal policy. Afterwards, we analytically demonstrate that
the structures of the age-optimal and throughput-optimal policies are
different. We also numerically demonstrate these structures as well as the
impact of system design parameters on the optimal achievable average weighted
sum-AoI.
