  We study the dynamics arising from a double quantum quench where the
parameters of a given Hamiltonian are abruptly changed from being in an
equilibrium phase A to a different phase B and back (A$\to$B$\to$A). As
prototype models, we consider the (integrable) transverse field Ising as well
as the (non-integrable) ANNNI model. The return amplitude features
non-analyticities after the first quench through the equilibrium quantum
critical point (A$\to$B), which is routinely taken as a signature of passing
through a so-called dynamical quantum phase transition. We demonstrate that
non-analyticities after the second quench (B$\to$A) can be avoided and
reestablished in a recurring manner upon increasing the time $T$ spent in phase
B. The system retains an infinite memory of its past state, and one has the
intriguing opportunity to control at will whether or not dynamical quantum
phase transitions appear after the second quench.
