  Background: Relevance to industry and scientific rigor have long been an area
of friction in IS research. However little work has been done on how to
evaluate IS research relevance. Kitchenham et al [13] proposed one of the few
relevance evaluating instruments in literature, later revised by Daneva et al
[7]. Aim: To analyze the practitioner/consultant perspective checklist1 for
relevance in order to evaluate its comprehensibility and applicability from the
point of view of the practitioner/consultant in the context of an advanced
university classroom. Method: Five master level students in the field of IS
assessed a set of 24 papers using the relevance checklist1. For each question
in the checklist, inter-rater agreement has been calculated and the reasoning
that the practitioners applied has been reconstructed from comments. Results:
Inter-rater agreement only showed to be slight for three questions and poor for
all other questions. Analysis of comments provided by the practitioners showed
only two questions that were interpreted in the same way by all practitioners.
These two questions showed significantly higher inter-rater agreement than
other questions. Conclusions: The generally low inter-rater agreement could be
explained as an indication that the checklist1 is in its current form not
appropriate for measuring industry relevance of IS research. The different
interpretations found for the checklist questions provide useful insight for
reformulation of questions. Reformulations are proposed for some questions.
