  The demand for efficient processing of deep neural networks (DNNs) on
embedded devices is a significant challenge limiting their deployment.
Exploiting sparsity in the network's feature maps is one of the ways to reduce
its inference latency. It is known that unstructured sparsity results in lower
accuracy degradation with respect to structured sparsity but the former needs
extensive inference engine changes to get latency benefits. To tackle this
challenge, we propose a solution to induce semi-structured activation sparsity
exploitable through minor runtime modifications. To attain high speedup levels
at inference time, we design a sparse training procedure with awareness of the
final position of the activations while computing the General Matrix
Multiplication (GEMM). We extensively evaluate the proposed solution across
various models for image classification and object detection tasks. Remarkably,
our approach yields a speed improvement of $1.25 \times$ with a minimal
accuracy drop of $1.1\%$ for the ResNet18 model on the ImageNet dataset.
Furthermore, when combined with a state-of-the-art structured pruning method,
the resulting models provide a good latency-accuracy trade-off, outperforming
models that solely employ structured pruning techniques.
