  Approximate Dynamic Programming (ADP) is a methodology to solve multi-stage
stochastic optimization problems in multi-dimensional discrete or continuous
spaces. ADP approximates the optimal value function by adaptively sampling both
action and state space. It provides a tractable approach to very large
problems, but can suffer from the exploration-exploitation dilemma. We propose
a novel approach for selecting actions using importance sampling weighted by
the value function approximation in continuous decision spaces to address this
dilemma. An advantage of this approach is it balances exploration and
exploitation without any tuning parameters when sampling actions compared to
other exploration approaches such as Epsilon Greedy, instead relying only on
the approximate value function. We compare the proposed algorithm with other
exploration strategies in continuous action space in the context of a
multi-stage generation expansion planning problem under uncertainty.
