  Users seek security & privacy (S&P) advice from online resources, including
trusted websites and content-sharing platforms. These resources help users
understand S&P technologies and tools and suggest actionable strategies. Large
Language Models (LLMs) have recently emerged as trusted information sources.
However, their accuracy and correctness have been called into question. Prior
research has outlined the shortcomings of LLMs in answering multiple-choice
questions and user ability to inadvertently circumvent model restrictions
(e.g., to produce toxic content). Yet, the ability of LLMs to provide reliable
S&P advice is not well-explored. In this paper, we measure their ability to
refute popular S&P misconceptions that the general public holds. We first study
recent academic literature to curate a dataset of over a hundred S&P-related
misconceptions across six different topics. We then query two popular LLMs
(Bard and ChatGPT) and develop a labeling guide to evaluate their responses to
these misconceptions. To comprehensively evaluate their responses, we further
apply three strategies: query each misconception multiple times, generate and
query their paraphrases, and solicit source URLs of the responses. Both models
demonstrate, on average, a 21.3% non-negligible error rate, incorrectly
supporting popular S&P misconceptions. The error rate increases to 32.6% when
we repeatedly query LLMs with the same or paraphrased misconceptions. We also
expose that models may partially support a misconception or remain
noncommittal, refusing a firm stance on misconceptions. Our exploration of
information sources for responses revealed that LLMs are susceptible to
providing invalid URLs (21.2% for Bard and 67.7% for ChatGPT) or point to
unrelated sources (44.2% returned by Bard and 18.3% by ChatGPT).
