  User comments have become an essential part of online journalism. However,
newsrooms are often overwhelmed by the vast number of diverse comments, for
which a manual analysis is barely feasible. Identifying meta-comments that
address or mention newsrooms, individual journalists, or moderators and that
may call for reactions is particularly critical. In this paper, we present an
automated approach to identify and classify meta-comments. We compare comment
classification based on manually extracted features with an end-to-end learning
approach. We develop, optimize, and evaluate multiple classifiers on a comment
dataset of the large German online newsroom SPIEGEL Online and the 'One Million
Posts' corpus of DER STANDARD, an Austrian newspaper. Both optimized
classification approaches achieved encouraging $F_{0.5}$ values between 76% and
91%. We report on the most significant classification features with the results
of a qualitative analysis and discuss how our work contributes to making
participation in online journalism more constructive.
