  We carry out a detailed analysis of the security advice coding method
(SAcoding) of Barrera et al. (2021), which is designed to analyze security
advice in the sense of measuring actionability and categorizing advice items as
practices, policies, principles, or outcomes. The main part of our analysis
explores the extent to which a second coder's assignment of codes to advice
items agrees with that of a first, for a dataset of 1013 security advice items
nominally addressing Internet of Things devices. More broadly, we seek a deeper
understanding of the soundness and utility of the SAcoding method, and the
degree to which it meets the design goal of reducing subjectivity in assigning
codes to security advice items. Our analysis results in suggestions for
modifications to the coding tree methodology, and some recommendations. We
believe the coding tree approach may be of interest for analysis of qualitative
data beyond security advice datasets alone.
