  Dynamical systems are often time-varying, whose modeling requires a function
that evolves with respect to time. Recent studies such as the neural ordinary
differential equation proposed a time-dependent neural network, which provides
a neural network varying with respect to time. However, we claim that the
architectural choice to build a time-dependent neural network significantly
affects its time-awareness but still lacks sufficient validation in its current
states. In this study, we conduct an in-depth analysis of the architecture of
modern time-dependent neural networks. Here, we report a vulnerability of
vanishing timestep embedding, which disables the time-awareness of a
time-dependent neural network. Furthermore, we find that this vulnerability can
also be observed in diffusion models because they employ a similar architecture
that incorporates timestep embedding to discriminate between different
timesteps during a diffusion process. Our analysis provides a detailed
description of this phenomenon as well as several solutions to address the root
cause. Through experiments on neural ordinary differential equations and
diffusion models, we observed that ensuring alive time-awareness via proposed
solutions boosted their performance, which implies that their current
implementations lack sufficient time-dependency.
