  To protect cryptographic implementations from side-channel vulnerabilities,
developers must adopt constant-time programming practices. As these can be
error-prone, many side-channel detection tools have been proposed. Despite
this, such vulnerabilities are still manually found in cryptographic libraries.
While a recent paper by Jancar et al. shows that developers rarely perform
side-channel detection, it is unclear if existing detection tools could have
found these vulnerabilities in the first place. To answer this question, we
surveyed the literature to build a classification of 34 side-channel detection
frameworks. The classification we offer compares multiple criteria, including
the methods used, the scalability of the analysis or the threat model
considered. We then built a unified common benchmark of representative
cryptographic operations on a selection of 5 promising detection tools. This
benchmark allows us to better compare the capabilities of each tool, and the
scalability of their analysis. Additionally, we offer a classification of
recently published side-channel vulnerabilities. We then test each of the
selected tools on benchmarks reproducing a subset of these vulnerabilities as
well as the context in which they appear. We find that existing tools can
struggle to find vulnerabilities for a variety of reasons, mainly the lack of
support for SIMD instructions, implicit flows, and internal secret generation.
Based on our findings, we develop a set of recommendations for the research
community and cryptographic library developers, with the goal to improve the
effectiveness of side-channel detection tools.
