  We investigate a Bayesian $k$-armed bandit problem in the \emph{many-armed}
regime, where $k \geq \sqrt{T}$ and $T$ represents the time horizon. Initially,
and aligned with recent literature on many-armed bandit problems, we observe
that subsampling plays a key role in designing optimal algorithms; the
conventional UCB algorithm is sub-optimal, whereas a subsampled UCB (SS-UCB),
which selects $\Theta(\sqrt{T})$ arms for execution under the UCB framework,
achieves rate-optimality. However, despite SS-UCB's theoretical promise of
optimal regret, it empirically underperforms compared to a greedy algorithm
that consistently chooses the empirically best arm. This observation extends to
contextual settings through simulations with real-world data. Our findings
suggest a new form of \emph{free exploration} beneficial to greedy algorithms
in the many-armed context, fundamentally linked to a tail event concerning the
prior distribution of arm rewards. This finding diverges from the notion of
free exploration, which relates to covariate variation, as recently discussed
in contextual bandit literature. Expanding upon these insights, we establish
that the subsampled greedy approach not only achieves rate-optimality for
Bernoulli bandits within the many-armed regime but also attains sublinear
regret across broader distributions. Collectively, our research indicates that
in the many-armed regime, practitioners might find greater value in adopting
greedy algorithms.
