  Edge sensing with micro-power pulse-Doppler radars is an emergent domain in
monitoring and surveillance with several smart city applications. Existing
solutions for the clutter versus multi-source radar classification task are
limited in terms of either accuracy or efficiency, and in some cases, struggle
with a trade-off between false alarms and recall of sources. We find that this
problem can be resolved by learning the classifier across multiple time-scales.
We propose a multi-scale, cascaded recurrent neural network architecture,
MSC-RNN, comprised of an efficient multi-instance learning (MIL) Recurrent
Neural Network (RNN) for clutter discrimination at a lower tier, and a more
complex RNN classifier for source classification at the upper tier. By
controlling the invocation of the upper RNN with the help of the lower tier
conditionally, MSC-RNN achieves an overall accuracy of 0.972. Our approach
holistically improves the accuracy and per-class recalls over ML models
suitable for radar inferencing. Notably, we outperform cross-domain handcrafted
feature engineering with time-domain deep feature learning, while also being up
to $\sim$3$\times$ more efficient than a competitive solution.
