  The real-time processing of time series signals is a critical issue for many
real-life applications. The idea of real-time processing is especially
important in audio domain as the human perception of sound is sensitive to any
kind of disturbance in perceived signals, especially the lag between auditory
and visual modalities. The rise of deep learning (DL) models complicated the
landscape of signal processing. Although they often have superior quality
compared to standard DSP methods, this advantage is diminished by higher
latency. In this work we propose novel method for minimization of inference
time latency and memory consumption, called Short-Term Memory Convolution
(STMC) and its transposed counterpart. The main advantage of STMC is the low
latency comparable to long short-term memory (LSTM) networks. Furthermore, the
training of STMC-based models is faster and more stable as the method is based
solely on convolutional neural networks (CNNs). In this study we demonstrate an
application of this solution to a U-Net model for a speech separation task and
GhostNet model in acoustic scene classification (ASC) task. In case of speech
separation we achieved a 5-fold reduction in inference time and a 2-fold
reduction in latency without affecting the output quality. The inference time
for ASC task was up to 4 times faster while preserving the original accuracy.
