  Data-intensive scientific and commercial applications increasingly require
frequent movement of large datasets from one site to the other(s). Despite
growing network capacities, these data movements rarely achieve the promised
data transfer rates of the underlying physical network due to poorly tuned data
transfer protocols. Accurately and efficiently tuning the data transfer
protocol parameters in a dynamically changing network environment is a major
challenge and remains as an open research problem. In this paper, we present
predictive end-to-end data transfer optimization algorithms based on historical
data analysis and real-time background traffic probing, dubbed HARP. Most of
the previous work in this area are solely based on real time network probing
which results either in an excessive sampling overhead or fails to accurately
predict the optimal transfer parameters. Combining historical data analysis
with real time sampling enables our algorithms to tune the application level
data transfer parameters accurately and efficiently to achieve close-to-optimal
end-to-end data transfer throughput with very low overhead. Our experimental
analysis over a variety of network settings shows that HARP outperforms
existing solutions by up to 50% in terms of the achieved throughput.
