  Within Reinforcement Learning, there is a growing collection of research
which aims to express all of an agent's knowledge of the world through
predictions about sensation, behaviour, and time. This work can be seen not
only as a collection of architectural proposals, but also as the beginnings of
a theory of machine knowledge in reinforcement learning. Recent work has
expanded what can be expressed using predictions, and developed applications
which use predictions to inform decision-making on a variety of synthetic and
real-world problems. While promising, we here suggest that the notion of
predictions as knowledge in reinforcement learning is as yet underdeveloped:
some work explicitly refers to predictions as knowledge, what the requirements
are for considering a prediction to be knowledge have yet to be well explored.
This specification of the necessary and sufficient conditions of knowledge is
important; even if claims about the nature of knowledge are left implicit in
technical proposals, the underlying assumptions of such claims have
consequences for the systems we design. These consequences manifest in both the
way we choose to structure predictive knowledge architectures, and how we
evaluate them. In this paper, we take a first step to formalizing predictive
knowledge by discussing the relationship of predictive knowledge learning
methods to existing theories of knowledge in epistemology. Specifically, we
explore the relationships between Generalized Value Functions and epistemic
notions of Justification and Truth.
