  Recently, automated co-design of machine learning (ML) models and accelerator
architectures has attracted significant attention from both the industry and
academia. However, most co-design frameworks either explore a limited search
space or employ suboptimal exploration techniques for simultaneous design
decision investigations of the ML model and the accelerator. Furthermore,
training the ML model and simulating the accelerator performance is
computationally expensive. To address these limitations, this work proposes a
novel neural architecture and hardware accelerator co-design framework, called
CODEBench. It is composed of two new benchmarking sub-frameworks, CNNBench and
AccelBench, which explore expanded design spaces of convolutional neural
networks (CNNs) and CNN accelerators. CNNBench leverages an advanced search
technique, BOSHNAS, to efficiently train a neural heteroscedastic surrogate
model to converge to an optimal CNN architecture by employing second-order
gradients. AccelBench performs cycle-accurate simulations for a diverse set of
accelerator architectures in a vast design space. With the proposed co-design
method, called BOSHCODE, our best CNN-accelerator pair achieves 1.4% higher
accuracy on the CIFAR-10 dataset compared to the state-of-the-art pair, while
enabling 59.1% lower latency and 60.8% lower energy consumption. On the
ImageNet dataset, it achieves 3.7% higher Top1 accuracy at 43.8% lower latency
and 11.2% lower energy consumption. CODEBench outperforms the state-of-the-art
framework, i.e., Auto-NBA, by achieving 1.5% higher accuracy and 34.7x higher
throughput, while enabling 11.0x lower energy-delay product (EDP) and 4.0x
lower chip area on CIFAR-10.
