  Dynamic mechanism design is a challenging extension to ordinary mechanism
design in which the mechanism designer must make a sequence of decisions over
time in the face of possibly untruthful reports of participating agents.
Optimizing dynamic mechanisms for welfare is relatively well understood.
However, there has been less work on optimizing for other goals (e.g. revenue),
and without restrictive assumptions on valuations, it is remarkably challenging
to characterize good mechanisms. Instead, we turn to automated mechanism design
to find mechanisms with good performance in specific problem instances. In
fact, the situation is similar even in static mechanism design. However, in the
static case, optimization/machine learning-based automated mechanism design
techniques have been successful in finding high-revenue mechanisms in cases
beyond the reach of analytical results. We extend the class of affine maximizer
mechanisms to MDPs where agents may untruthfully report their rewards. This
extension results in a challenging bilevel optimization problem in which the
upper problem involves choosing optimal mechanism parameters, and the lower
problem involves solving the resulting MDP. Our approach can find truthful
dynamic mechanisms that achieve strong performance on goals other than welfare,
and can be applied to essentially any problem setting-without restrictions on
valuations-for which RL can learn optimal policies.
