  Black-box nature of Artificial Intelligence (AI) models do not allow users to
comprehend and sometimes trust the output created by such model. In AI
applications, where not only the results but also the decision paths to the
results are critical, such black-box AI models are not sufficient. Explainable
Artificial Intelligence (XAI) addresses this problem and defines a set of AI
models that are interpretable by the users. Recently, several number of XAI
models have been to address the issues surrounding by lack of interpretability
and explainability of black-box models in various application areas such as
healthcare, military, energy, financial and industrial domains. Although the
concept of XAI has gained great deal of attention recently, its integration
into the IoT domain has not yet been fully defined. In this paper, we provide
an in-depth and systematic review of recent studies using XAI models in the
scope of IoT domain. We categorize the studies according to their methodology
and applications areas. In addition, we aim to focus on the challenging
problems and open issues and give future directions to guide the developers and
researchers for prospective future investigations.
