  Fault diagnosis of dynamic systems is done by detecting changes in
time-series data, for example residuals, caused by system degradation and
faulty components. The use of general-purpose multi-class classification
methods for fault diagnosis is complicated by imbalanced training data and
unknown fault classes. Another complicating factor is that different fault
classes can result in similar residual outputs, especially for small faults,
which causes classification ambiguities. In this work, a framework for
data-driven analysis and open-set classification is developed for fault
diagnosis applications using the Kullback-Leibler divergence. A data-driven
fault classification algorithm is proposed which can handle imbalanced
datasets, class overlapping, and unknown faults. In addition, an algorithm is
proposed to estimate the size of the fault when training data contains
information from known fault realizations. An advantage of the proposed
framework is that it can also be used for quantitative analysis of fault
diagnosis performance, for example, to analyze how easy it is to classify
faults of different magnitudes. To evaluate the usefulness of the proposed
methods, multiple datasets from different fault scenarios have been collected
from an internal combustion engine test bench to illustrate the design process
of a data-driven diagnosis system, including quantitative fault diagnosis
analysis and evaluation of the developed open set fault classification
algorithm.
