  Timing-based side and covert channels in processor caches continue to be a
threat to modern computers. This work shows for the first time a systematic,
large-scale analysis of Arm devices and the detailed results of attacks the
processors are vulnerable to. Compared to x86, Arm uses different
architectures, microarchitectural implementations, cache replacement policies,
etc., which affects how attacks can be launched, and how security testing for
the vulnerabilities should be done. To evaluate security, this paper presents
security benchmarks specifically developed for testing Arm processors and their
caches. The benchmarks are themselves evaluated with sensitivity tests, which
examine how sensitive the benchmarks are to having a correct configuration in
the testing phase. Further, to evaluate a large number of devices, this work
leverages a novel approach of using a cloud-based Arm device testbed for
architectural and security research on timing channels and runs the benchmarks
on 34 different physical devices. In parallel, there has been much interest in
secure caches to defend the various attacks. Consequently, this paper also
investigates secure cache architectures using the proposed benchmarks.
Especially, this paper implements and evaluates the secure PL and RF caches,
showing the security of PL and RF caches, but also uncovers new weaknesses.
