  Multiplication is arguably the most cost-dominant operation in modern deep
neural networks (DNNs), limiting their achievable efficiency and thus more
extensive deployment in resource-constrained applications. To tackle this
limitation, pioneering works have developed handcrafted multiplication-free
DNNs, which require expert knowledge and time-consuming manual iteration,
calling for fast development tools. To this end, we propose a Neural
Architecture Search and Acceleration framework dubbed NASA, which enables
automated multiplication-reduced DNN development and integrates a dedicated
multiplication-reduced accelerator for boosting DNNs' achievable efficiency.
Specifically, NASA adopts neural architecture search (NAS) spaces that augment
the state-of-the-art one with hardware-inspired multiplication-free operators,
such as shift and adder, armed with a novel progressive pretrain strategy (PGP)
together with customized training recipes to automatically search for optimal
multiplication-reduced DNNs; On top of that, NASA further develops a dedicated
accelerator, which advocates a chunk-based template and auto-mapper dedicated
for NASA-NAS resulting DNNs to better leverage their algorithmic properties for
boosting hardware efficiency. Experimental results and ablation studies
consistently validate the advantages of NASA's algorithm-hardware co-design
framework in terms of achievable accuracy and efficiency tradeoffs. Codes are
available at https://github.com/GATECH-EIC/NASA.
