  In Fog-assisted IoT systems, it is a common practice to cache popular content
at the network edge to achieve high quality of service. Due to uncertainties in
practice such as unknown file popularities, cache placement scheme design is
still an open problem with unresolved challenges: 1) how to maintain
time-averaged storage costs under budgets, 2) how to incorporate online
learning to aid cache placement to minimize performance loss (a.k.a. regret),
and 3) how to exploit offline historical information to further reduce regret.
In this paper, we formulate the cache placement problem with unknown file
popularities as a constrained combinatorial multi-armed bandit (CMAB) problem.
To solve the problem, we employ virtual queue techniques to manage
time-averaged storage cost constraints, and adopt history-aware bandit learning
methods to integrate offline historical information into the online learning
procedure to handle the exploration-exploitation tradeoff. With an effective
combination of online control and history-aware online learning, we devise a
Cache Placement scheme with History-aware Bandit Learning called CPHBL. Our
theoretical analysis and simulations show that CPHBL achieves a sublinear
time-averaged regret bound. Moreover, the simulation results verify CPHBL's
advantage over the deep reinforcement learning based approach.
