  By leveraging the concept of mobile edge computing (MEC), massive amount of
data generated by a large number of Internet of Things (IoT) devices could be
offloaded to MEC server at the edge of wireless network for further
computational intensive processing. However, due to the resource constraint of
IoT devices and wireless network, both the communications and computation
resources need to be allocated and scheduled efficiently for better system
performance. In this paper, we propose a joint computation offloading and
multi-user scheduling algorithm for IoT edge computing system to minimize the
long-term average weighted sum of delay and power consumption under stochastic
traffic arrival. We formulate the dynamic optimization problem as an
infinite-horizon average-reward continuous-time Markov decision process (CTMDP)
model. One critical challenge in solving this MDP problem for the multi-user
resource control is the curse-of-dimensionality problem, where the state space
of the MDP model and the computation complexity increase exponentially with the
growing number of users or IoT devices. In order to overcome this challenge, we
use the deep reinforcement learning (RL) techniques and propose a neural
network architecture to approximate the value functions for the post-decision
system states. The designed algorithm to solve the CTMDP problem supports
semi-distributed auction-based implementation, where the IoT devices submit
bids to the BS to make the resource control decisions centrally. Simulation
results show that the proposed algorithm provides significant performance
improvement over the baseline algorithms, and also outperforms the RL
algorithms based on other neural network architectures.
