  The Common Vulnerability Scoring System (CVSS) is a popular method for
evaluating the severity of vulnerabilities in vulnerability management. In the
evaluation process, a numeric score between 0 and 10 is calculated, 10 being
the most severe (critical) value. The goal of CVSS is to provide comparable
scores across different evaluators. However, previous works indicate that CVSS
might not reach this goal: If a vulnerability is evaluated by several analysts,
their scores often differ. This raises the following questions: Are CVSS
evaluations consistent? Which factors influence CVSS assessments? We
systematically investigate these questions in an online survey with 196 CVSS
users. We show that specific CVSS metrics are inconsistently evaluated for
widespread vulnerability types, including Top 3 vulnerabilities from the "2022
CWE Top 25 Most Dangerous Software Weaknesses" list. In a follow-up survey with
59 participants, we found that for the same vulnerabilities from the main
study, 68% of these users gave different severity ratings. Our study reveals
that most evaluators are aware of the problematic aspects of CVSS, but they
still see CVSS as a useful tool for vulnerability assessment. Finally, we
discuss possible reasons for inconsistent evaluations and provide
recommendations on improving the consistency of scoring.
