  The applications of Artificial Intelligence (AI) methods especially machine
learning techniques have increased in recent years. Classification algorithms
have been successfully applied to different problems such as requirement
classification. Although these algorithms have good performance, most of them
cannot explain how they make a decision. Explainable Artificial Intelligence
(XAI) is a set of new techniques that explain the predictions of machine
learning algorithms. In this work, the applicability of XAI for software
requirement classification is studied. An explainable software requirement
classifier is presented using the LIME algorithm. The explainability of the
proposed method is studied by applying it to the PROMISE software requirement
dataset. The results show that XAI can help the analyst or requirement
specifier to better understand why a specific requirement is classified as
functional or non-functional. The important keywords for such decisions are
identified and analyzed in detail. The experimental study shows that the XAI
can be used to help analysts and requirement specifiers to better understand
the predictions of the classifiers for categorizing software requirements.
Also, the effect of the XAI on feature reduction is analyzed. The results
showed that the XAI model has a positive role in feature analysis.
