  With the growing demand for deploying deep learning models to the "edge", it
is paramount to develop techniques that allow to execute state-of-the-art
models within very tight and limited resource constraints. In this work we
propose a software-hardware optimization paradigm for obtaining a highly
efficient execution engine of deep neural networks (DNNs) that are based on
fully-connected layers. Our approach is centred around compression as a means
for reducing the area as well as power requirements of, concretely, multilayer
perceptrons (MLPs) with high predictive performances. Firstly, we design a
novel hardware architecture named FantastIC4, which (1) supports the efficient
on-chip execution of multiple compact representations of fully-connected layers
and (2) minimizes the required number of multipliers for inference down to only
4 (thus the name). Moreover, in order to make the models amenable for efficient
execution on FantastIC4, we introduce a novel entropy-constrained training
method that renders them to be robust to 4bit quantization and highly
compressible in size simultaneously. The experimental results show that we can
achieve throughputs of 2.45 TOPS with a total power consumption of 3.6W on a
Virtual Ultrascale FPGA XCVU440 device implementation, and achieve a total
power efficiency of 20.17 TOPS/W on a 22nm process ASIC version. When compared
to the other state-of-the-art accelerators designed for the Google Speech
Command (GSC) dataset, FantastIC4 is better by 51$\times$ in terms of
throughput and 145$\times$ in terms of area efficiency (GOPS/W).
