  We propose a Similarity-Based Stratified Splitting (SBSS) technique, which
uses both the output and input space information to split the data. The splits
are generated using similarity functions among samples to place similar samples
in different splits. This approach allows for a better representation of the
data in the training phase. This strategy leads to a more realistic performance
estimation when used in real-world applications. We evaluate our proposal in
twenty-two benchmark datasets with classifiers such as Multi-Layer Perceptron,
Support Vector Machine, Random Forest and K-Nearest Neighbors, and five
similarity functions Cityblock, Chebyshev, Cosine, Correlation, and Euclidean.
According to the Wilcoxon Sign-Rank test, our approach consistently
outperformed ordinary stratified 10-fold cross-validation in 75\% of the
assessed scenarios.
