  The applications of artificial intelligence (AI) are rapidly evolving, and
they are also commonly used in safety-critical domains, such as autonomous
driving and medical diagnosis, where functional safety is paramount. In
AI-driven systems, uncertainty estimation allows the user to avoid
overconfidence predictions and achieve functional safety. Therefore, the
robustness and reliability of model predictions can be improved. However,
conventional uncertainty estimation methods, such as the deep ensemble method,
impose high computation and, accordingly, hardware (latency and energy)
overhead because they require the storage and processing of multiple models.
Alternatively, Monte Carlo dropout (MC-dropout) methods, although having low
memory overhead, necessitate numerous ($\sim 100$) forward passes, leading to
high computational overhead and latency. Thus, these approaches are not
suitable for battery-powered edge devices with limited computing and memory
resources. In this paper, we propose the Tiny-Deep Ensemble approach, a
low-cost approach for uncertainty estimation on edge devices. In our approach,
only normalization layers are ensembled $M$ times, with all ensemble members
sharing common weights and biases, leading to a significant decrease in storage
requirements and latency. Moreover, our approach requires only one forward pass
in a hardware architecture that allows batch processing for inference and
uncertainty estimation. Furthermore, it has approximately the same memory
overhead compared to a single model. Therefore, latency and memory overhead are
reduced by a factor of up to $\sim M\times$. Nevertheless, our method does not
compromise accuracy, with an increase in inference accuracy of up to $\sim 1\%$
and a reduction in RMSE of $17.17\%$ in various benchmark datasets, tasks, and
state-of-the-art architectures.
