  Foundation models often struggle with uncertainty when faced with new
situations in online decision-making, necessitating scalable and efficient
exploration to resolve this uncertainty. We introduce GPT-HyperAgent, an
augmentation of GPT with HyperAgent for uncertainty-aware, scalable exploration
in contextual bandits, a fundamental online decision problem involving natural
language input. We prove that HyperAgent achieves fast incremental uncertainty
estimation with $\tilde{O}(\log T)$ per-step computational complexity over $T$
periods under the linear realizable assumption. Our analysis demonstrates that
HyperAgent's regret order matches that of exact Thompson sampling in linear
contextual bandits, closing a significant theoretical gap in scalable
exploration. Empirical results in real-world contextual bandit tasks, such as
automated content moderation with human feedback, validate the practical
effectiveness of GPT-HyperAgent for safety-critical decisions. Our code is
open-sourced at \url{https://github.com/szrlee/GPT-HyperAgent/}.
