  We introduce Cloud DIKW as an analysis environment supporting scientific
discovery through integrated parallel batch and streaming processing, and apply
it to one representative domain application: social media data stream
clustering. Recent work demonstrated that high-quality clusters can be
generated by representing the data points using high-dimensional vectors that
reflect textual content and social network information. Due to the high cost of
similarity computation, sequential implementations of even single-pass
algorithms cannot keep up with the speed of real-world streams. This paper
presents our efforts to meet the constraints of real-time social stream
clustering through parallelization. We focus on two system-level issues. Most
stream processing engines like Apache Storm organize distributed workers in the
form of a directed acyclic graph, making it difficult to dynamically
synchronize the state of parallel workers. We tackle this challenge by creating
a separate synchronization channel using a pub-sub messaging system. Due to the
sparsity of the high-dimensional vectors, the size of centroids grows quickly
as new data points are assigned to the clusters. Traditional synchronization
that directly broadcasts cluster centroids becomes too expensive and limits the
scalability of the parallel algorithm. We address this problem by communicating
only dynamic changes of the clusters rather than the whole centroid vectors.
Our algorithm under Cloud DIKW can process the Twitter 10% data stream in
real-time with 96-way parallelism. By natural improvements to Cloud DIKW,
including advanced collective communication techniques developed in our Harp
project, we will be able to process the full Twitter stream in real-time with
1000-way parallelism. Our use of powerful general software subsystems will
enable many other applications that need integration of streaming and batch
data analytics.
