  Recent development in computing, sensing and crowd-sourced data have resulted
in an explosion in the availability of quantitative information. The
possibilities of analyzing this so-called Big Data to inform research and the
decision-making process are virtually endless. In general, analyses have to be
done across multiple data sets in order to bring out the most value of Big
Data. A first important step is to identify temporal correlations between data
sets. Given the characteristics of Big Data in terms of volume and velocity,
techniques that identify correlations not only need to be fast and scalable,
but also need to help users in ordering the correlations across temporal scales
so that they can focus on important relationships. In this paper, we present
AMIC (Adaptive Mutual Information-based Correlation), a method based on mutual
information to identify correlations at multiple temporal scales in large time
series. Discovered correlations are suggested to users in an order based on the
strength of the relationships. Our method supports an adaptive streaming
technique that minimizes duplicated computation and is implemented on top of
Apache Spark for scalability. We also provide a comprehensive evaluation on the
effectiveness and the scalability of AMIC using both synthetic and real-world
data sets.
