  Estimating optimal dynamic policies from offline data is a fundamental
problem in dynamic decision making. In the context of causal inference, the
problem is known as estimating the optimal dynamic treatment regime. Even
though there exists a plethora of methods for estimation, constructing
confidence intervals for the value of the optimal regime and structural
parameters associated with it is inherently harder, as it involves non-linear
and non-differentiable functionals of unknown quantities that need to be
estimated. Prior work resorted to sub-sample approaches that can deteriorate
the quality of the estimate. We show that a simple soft-max approximation to
the optimal treatment regime, for an appropriately fast growing temperature
parameter, can achieve valid inference on the truly optimal regime. We
illustrate our result for a two-period optimal dynamic regime, though our
approach should directly extend to the finite horizon case. Our work combines
techniques from semi-parametric inference and $g$-estimation, together with an
appropriate triangular array central limit theorem, as well as a novel analysis
of the asymptotic influence and asymptotic bias of softmax approximations.
