  PV power forecasting models are predominantly based on machine learning
algorithms which do not provide any insight into or explanation about their
predictions (black boxes). Therefore, their direct implementation in
environments where transparency is required, and the trust associated with
their predictions may be questioned. To this end, we propose a two stage
probabilistic forecasting framework able to generate highly accurate, reliable,
and sharp forecasts yet offering full transparency on both the point forecasts
and the prediction intervals (PIs). In the first stage, we exploit natural
gradient boosting (NGBoost) for yielding probabilistic forecasts, while in the
second stage, we calculate the Shapley additive explanation (SHAP) values in
order to fully comprehend why a prediction was made. To highlight the
performance and the applicability of the proposed framework, real data from two
PV parks located in Southern Germany are employed. Comparative results with two
state-of-the-art algorithms, namely Gaussian process and lower upper bound
estimation, manifest a significant increase in the point forecast accuracy and
in the overall probabilistic performance. Most importantly, a detailed analysis
of the model's complex nonlinear relationships and interaction effects between
the various features is presented. This allows interpreting the model,
identifying some learned physical properties, explaining individual
predictions, reducing the computational requirements for the training without
jeopardizing the model accuracy, detecting possible bugs, and gaining trust in
the model. Finally, we conclude that the model was able to develop complex
nonlinear relationships which follow known physical properties as well as human
logic and intuition.
