  This work applies reinforcement learning (RL) from the AI machine learning
field to derive an optimal Bitcoin-like blockchain mining strategy without
knowing the details of the blockchain network model. Previously, the most
profitable mining strategy was believed to be honest mining encoded in the
default blockchain protocol. It was shown later that it is possible to gain
more mining rewards by deviating from honest mining. In particular, the mining
problem can be formulated as a Markov Decision Process (MDP) which can be
solved to give the optimal mining strategy. However, solving the mining MDP
requires knowing the values of various parameters that characterize the
blockchain network model. In real blockchain networks, these parameter values
are not easy to obtain and may change over time. This hinders the use of the
MDP model-based solution. In this work, we employ RL to dynamically learn a
mining strategy with performance approaching that of the optimal mining
strategy by observing and interacting with the network. Since the mining MDP
problem has a non-linear objective function (rather than linear functions of
standard MDP problems), we design a new multi-dimensional RL algorithm to solve
the problem. Experimental results indicate that, without knowing the parameter
values of the mining MDP model, our multi-dimensional RL mining algorithm can
still achieve the optimal performance over time-varying blockchain networks.
