  We propose an algorithm for next query recommendation in interactive data
exploration settings, like knowledge discovery for information gathering. The
state-of-the-art query recommendation algorithms are based on
sequence-to-sequence learning approaches that exploit historical interaction
data. Due to the supervision involved in the learning process, such approaches
fail to adapt to immediate user feedback. We propose to augment the
transformer-based causal language models for query recommendations to adapt to
the immediate user feedback using multi-armed bandit (MAB) framework. We
conduct a large-scale experimental study using log files from a popular online
literature discovery service and demonstrate that our algorithm improves the
per-round regret substantially, with respect to the state-of-the-art
transformer-based query recommendation models, which do not make use of
immediate user feedback. Our data model and source code are available at
https://github.com/shampp/exp3_ss
