  The machine learning modeling process conventionally culminates in selecting
a single model that maximizes a selected performance metric. However, this
approach leads to abandoning a more profound analysis of slightly inferior
models. Particularly in medical and healthcare studies, where the objective
extends beyond predictions to valuable insight generation, relying solely on a
single model can result in misleading or incomplete conclusions. This problem
is particularly pertinent when dealing with a set of models known as
$\textit{Rashomon set}$, with performance close to maximum one. Such a set can
be numerous and may contain models describing the data in a different way,
which calls for comprehensive analysis. This paper introduces a novel process
to explore models in the Rashomon set, extending the conventional modeling
approach. We propose the $\texttt{Rashomon_DETECT}$ algorithm to detect models
with different behavior. It is based on recent developments in the eXplainable
Artificial Intelligence (XAI) field. To quantify differences in variable
effects among models, we introduce the Profile Disparity Index (PDI) based on
measures from functional data analysis. To illustrate the effectiveness of our
approach, we showcase its application in predicting survival among
hemophagocytic lymphohistiocytosis (HLH) patients - a foundational case study.
Additionally, we benchmark our approach on other medical data sets,
demonstrating its versatility and utility in various contexts. If differently
behaving models are detected in the Rashomon set, their combined analysis leads
to more trustworthy conclusions, which is of vital importance for high-stakes
applications such as medical applications.
