  Altmetrics have been proposed as a way to assess the societal impact of
research. Although altmetrics are already in use as impact or attention metrics
in different contexts, it is still not clear whether they really capture or
reflect societal impact. This study is based on altmetrics, citation counts,
research output and case study data from the UK Research Excellence Framework
(REF), and peers' REF assessments of research output and societal impact. We
investigated the convergent validity of altmetrics by using two REF datasets:
publications submitted as research output (PRO) to the REF and publications
referenced in case studies (PCS). Case studies, which are intended to
demonstrate societal impact, should cite the most relevant research papers. We
used the MHq' indicator for assessing impact - an indicator which has been
introduced for count data with many zeros. The results of the first part of the
analysis show that news media as well as mentions on Facebook, in blogs, in
Wikipedia, and in policy-related documents have higher MHq' values for PCS than
for PRO. Thus, the altmetric indicators seem to have convergent validity for
these data. In the second part of the analysis, altmetrics have been correlated
with REF reviewers' average scores on PCS. The negative or close to zero
correlations question the convergent validity of altmetrics in that context. We
suggest that they may capture a different aspect of societal impact (which can
be called unknown attention) to that seen by reviewers (who are interested in
the causal link between research and action in society).
