  Diagnosing and fixing performance problems on multicore machines with deep
memory hierarchies is extremely challenging. Certain problems are best
addressed when we can analyze the entire trace of program execution, e.g.,
every memory access. Unfortunately such detailed execution logs are very large
and cannot be analyzed by direct inspection. We present DINAMITE: a toolkit for
Dynamic INstrumentation and Analysis for MassIve Trace Exploration. DINAMITE is
a collection of tools for end-to-end performance analysis: from the LLVM
compiler pass that instruments the program to plug-and-play tools that use a
modern data analytics engine Spark Streaming for trace introspection. Using
DINAMITE we found opportunities to improve data layout in several applications
that resulted in 15-20% performance improvements and found a shared-variable
bottleneck in a popular key-value store, whose elimination improved performance
by 20x.
