  The reliability of deep time series models is often compromised by their
tendency to rely on confounding factors, which may lead to incorrect outputs.
Our newly recorded, naturally confounded dataset named P2S from a real
mechanical production line emphasizes this. To avoid "Clever-Hans" moments in
time series, i.e., to mitigate confounders, we introduce the method Right on
Time (RioT). RioT enables, for the first time, interactions with model
explanations across both the time and frequency domain. Feedback on
explanations in both domains is then used to constrain the model, steering it
away from the annotated confounding factors. The dual-domain interaction
strategy is crucial for effectively addressing confounders in time series
datasets. We empirically demonstrate that RioT can effectively guide models
away from the wrong reasons in P2S as well as popular time series
classification and forecasting datasets.
