  We study reserve price optimization in multi-phase second price auctions,
where seller's prior actions affect the bidders' later valuations through a
Markov Decision Process (MDP). Compared to the bandit setting in existing
works, the setting in ours involves three challenges. First, from the seller's
perspective, we need to efficiently explore the environment in the presence of
potentially nontruthful bidders who aim to manipulates seller's policy. Second,
we want to minimize the seller's revenue regret when the market noise
distribution is unknown. Third, the seller's per-step revenue is unknown,
nonlinear, and cannot even be directly observed from the environment.
  We propose a mechanism addressing all three challenges. To address the first
challenge, we use a combination of a new technique named "buffer periods" and
inspirations from Reinforcement Learning (RL) with low switching cost to limit
bidders' surplus from untruthful bidding, thereby incentivizing approximately
truthful bidding. The second one is tackled by a novel algorithm that removes
the need for pure exploration when the market noise distribution is unknown.
The third challenge is resolved by an extension of LSVI-UCB, where we use the
auction's underlying structure to control the uncertainty of the revenue
function. The three techniques culminate in the $\underline{\rm
C}$ontextual-$\underline{\rm L}$SVI-$\underline{\rm U}$CB-$\underline{\rm
B}$uffer (CLUB) algorithm which achieves $\tilde{
\mathcal{O}}(H^{5/2}\sqrt{K})$ revenue regret when the market noise is known
and $\tilde{ \mathcal{O}}(H^{3}\sqrt{K})$ revenue regret when the noise is
unknown with no assumptions on bidders' truthfulness.
