  In this paper, we consider a new Multi-Armed Bandit (MAB) problem where arms
are nodes in an unknown and possibly changing graph, and the agent (i)
initiates random walks over the graph by pulling arms, (ii) observes the random
walk trajectories, and (iii) receives rewards equal to the lengths of the
walks. We provide a comprehensive understanding of this problem by studying
both the stochastic and the adversarial setting. We show that this problem is
not easier than a standard MAB in an information theoretical sense, although
additional information is available through random walk trajectories. Behaviors
of bandit algorithms on this problem are also studied.
