  A sequential decision-making agent balances between exploring to gain new
knowledge about an environment and exploiting current knowledge to maximize
immediate reward. For environments studied in the traditional literature,
optimal decisions gravitate over time toward exploitation as the agent
accumulates sufficient knowledge and the benefits of further exploration
vanish. What if, however, the environment offers an unlimited amount of useful
knowledge and there is large benefit to further exploration no matter how much
the agent has learned? We offer a simple, quintessential example of such a
complex environment. In this environment, rewards are unbounded and an agent
can always increase the rate at which rewards accumulate by exploring to learn
more. Consequently, an optimal agent forever maintains a propensity to explore.
