  In this paper we introduce a new approach to discrete-time semi-Markov
decision processes based on the sojourn time process. Different
characterizations of discrete-time semi-Markov processes are exploited and
decision processes are constructed by their means. With this new approach, the
agent is allowed to consider different actions depending also on the sojourn
time of the process in the current state. A numerical method based on
$Q$-learning algorithms for finite horizon reinforcement learning and
stochastic recursive relations is investigated. Finally, we consider two toy
examples: one in which the reward depends on the sojourn-time, according to the
gambler's fallacy; the other in which the environment is semi-Markov even if
the reward function does not depend on the sojourn time. These are used to
carry on some numerical evaluations on the previously presented $Q$-learning
algorithm and on a different naive method based on deep reinforcement learning.
