  In this work, a new clustering algorithm is proposed based on neutrosophic
set (NS) theory. The main contribution is to use NS to handle boundary and
outlier points as challenging points of clustering methods. In the first step,
a new definition of data indeterminacy (indeterminacy set) is proposed in NS
domain based on density properties of data. Lower indeterminacy is assigned to
data points in dense regions and vice versa. In the second step, indeterminacy
set is presented for a proposed cost function in NS domain by considering a set
of main clusters and a noisy cluster. In the proposed cost function, two
conditions based on distance from cluster centers and value of indeterminacy,
are considered for each data point. In the third step, the proposed cost
function is minimized by gradient descend methods. Data points are clustered
based on their membership degrees. Outlier points are assigned to noise
cluster; and boundary points are assigned to main clusters with almost same
membership degrees. To show the effectiveness of the proposed method, three
types of datasets including diamond, UCI and image datasets are used. Results
demonstrate that the proposed cost function handles boundary and outlier points
with more accurate membership degrees and outperforms existing state of the art
clustering methods in all datasets.
