  Recently, in the area of big data, some popular applications such as web
search engines and recommendation systems, face the problem to diversify
results during query processing. In this sense, it is both significant and
essential to propose methods to deal with big data in order to increase the
diversity of the result set. In this paper, we firstly define a set's diversity
and an element's ability to improve the set's overall diversity. Based on these
definitions, we propose a diversification framework which has good performance
in terms of effectiveness and efficiency. Also, this framework has theoretical
guarantee on probability of success. Secondly, we design implementation
algorithms based on this framework for both numerical and string data. Thirdly,
for numerical and string data respectively, we carry out extensive experiments
on real data to verify the performance of our proposed framework, and also
perform scalability experiments on synthetic data.
