  Distributed approaches based on the map-reduce programming paradigm have
started to be proposed in the bioinformatics domain, due to the large amount of
data produced by the next-generation sequencing techniques. However, the use of
map-reduce and related Big Data technologies and frameworks (e.g., Apache
Hadoop and Spark) does not necessarily produce satisfactory results, in terms
of both efficiency and effectiveness. We discuss how the development of
distributed and Big Data management technologies has affected the analysis of
large datasets of biological sequences. Moreover, we show how the choice of
different parameter configurations and the careful engineering of the software
with respect to the specific framework under consideration may be crucial in
order to achieve good performance, especially on very large amounts of data. We
choose k-mers counting as a case study for our analysis, and Spark as the
framework to implement FastKmer, a novel approach for the extraction of k-mer
statistics from large collection of biological sequences, with arbitrary values
of k. One of the most relevant contributions of FastKmer is the introduction of
a module for balancing the statistics aggregation workload over the nodes of a
computing cluster, in order to overcome data skew while allowing for a fully
exploitation of the underly- ing distributed architecture. We also present the
results of a comparative experimental analysis showing that our approach is
currently the fastest among the ones based on Big Data technologies, while
exhibiting a very good scalability. We provide evidence that the usage of
technologies such as Hadoop or Spark for the analysis of big datasets of
biological sequences is productive only if the architectural details and the
peculiar aspects of the considered framework are carefully taken into account
for the algorithm design and implementation.
