  We consider a classical and possibly driven composite system $X \otimes Y$
weakly coupled to a Markovian thermal reservoir $R$ so that an unambiguous
stochastic thermodynamics ensues for $X \otimes Y$. This setup can be
equivalently seen as a system $X$ strongly coupled to a non-Markovian reservoir
$Y \otimes R$. We demonstrate that only in the limit where the dynamics of $Y$
is much faster then $X$, our unambiguous expressions for thermodynamic
quantities such as heat, entropy or internal energy, are equivalent to the
strong coupling expressions recently obtained in the literature using the
Hamiltonian of mean force. By doing so, we also significantly extend these
results by formulating them at the level of instantaneous rates and by allowing
for time-dependent couplings between $X$ and its environment. Away from the
limit where $Y$ evolves much faster than $X$, previous approaches fail to
reproduce the correct results from the original unambiguous formulation, as we
illustrate numerically for an underdamped Brownian particle coupled strongly to
a non-Markovian reservoir.
