  Successful applications of reinforcement learning in real-world problems
often require dealing with partially observable states. It is in general very
challenging to construct and infer hidden states as they often depend on the
agent's entire interaction history and may require substantial domain
knowledge. In this work, we investigate a deep-learning approach to learning
the representation of states in partially observable tasks, with minimal prior
knowledge of the domain. In particular, we propose a new family of hybrid
models that combines the strength of both supervised learning (SL) and
reinforcement learning (RL), trained in a joint fashion: The SL component can
be a recurrent neural networks (RNN) or its long short-term memory (LSTM)
version, which is equipped with the desired property of being able to capture
long-term dependency on history, thus providing an effective way of learning
the representation of hidden states. The RL component is a deep Q-network (DQN)
that learns to optimize the control for maximizing long-term rewards. Extensive
experiments in a direct mailing campaign problem demonstrate the effectiveness
and advantages of the proposed approach, which performs the best among a set of
previous state-of-the-art methods.
