  Single computation engines have become a popular design choice for FPGA-based
convolutional neural networks (CNNs) enabling the deployment of diverse models
without fabric reconfiguration. This flexibility, however, often comes with
significantly reduced performance on memory-bound layers and resource
underutilisation due to suboptimal mapping of certain layers on the engine's
fixed configuration. In this work, we investigate the implications in terms of
CNN engine design for a class of models that introduce a pre-convolution stage
to decompress the weights at run time. We refer to these approaches as
on-the-fly. To minimise the negative impact of limited bandwidth on
memory-bound layers, we present a novel hardware component that enables the
on-chip on-the-fly generation of weights. We further introduce an input
selective processing element (PE) design that balances the load between PEs on
suboptimally mapped layers. Finally, we present unzipFPGA, a framework to train
on-the-fly models and traverse the design space to select the highest
performing CNN engine configuration. Quantitative evaluation shows that
unzipFPGA yields an average speedup of 2.14x and 71% over optimised status-quo
and pruned CNN engines under constrained bandwidth and up to 3.69x higher
performance density over the state-of-the-art FPGA-based CNN accelerators.
