  Active inference is a theory of perception, learning and decision making,
which can be applied to neuroscience, robotics, and machine learning. Recently,
reasearch has been taking place to scale up this framework using Monte-Carlo
tree search and deep learning. The goal of this activity is to solve more
complicated tasks using deep active inference. First, we review the existing
literature, then, we progresively build a deep active inference agent. For two
agents, we have experimented with five definitions of the expected free energy
and three different action selection strategies. According to our experiments,
the models able to solve the dSprites environment are the ones that maximise
rewards. Finally, we compare the similarity of the representation learned by
the layers of various agents using centered kernel alignment. Importantly, the
agent maximising reward and the agent minimising expected free energy learn
very similar representations except for the last layer of the critic network
(reflecting the difference in learning objective), and the variance layers of
the transition and encoder networks. We found that the reward maximising agent
is a lot more certain than the agent minimising expected free energy. This is
because the agent minimising expected free energy always picks the action down,
and does not gather enough data for the other actions. In contrast, the agent
maximising reward, keeps on selecting the actions left and right, enabling it
to successfully solve the task. The only difference between those two agents is
the epistemic value, which aims to make the outputs of the transition and
encoder networks as close as possible. Thus, the agent minimising expected free
energy picks a single action (down), and becomes an expert at predicting the
future when selecting this action. This makes the KL divergence between the
output of the transition and encoder networks small.
