  Compiler optimization decisions are often based on hand-crafted heuristics
centered around a few established benchmark suites. Alternatively, they can be
learned from feature and performance data produced during compilation. However,
data-driven compiler optimizations based on machine learning models require
large sets of quality data for training in order to match or even outperform
existing human-crafted heuristics. In static compilation setups, related work
has addressed this problem with iterative compilation. However, a dynamic
compiler may produce different data depending on dynamically-chosen compilation
strategies, which aggravates the generation of comparable data. We propose
compilation forking, a technique for generating consistent feature and
performance data from arbitrary, dynamically-compiled programs. Different
versions of program parts with the same profiling and compilation history are
executed within single program runs to minimize noise stemming from dynamic
compilation and the runtime environment. Our approach facilitates large-scale
performance evaluations of compiler optimization decisions. Additionally,
compilation forking supports creating domain-specific compilation strategies
based on machine learning by providing the data for model training. We
implemented compilation forking in the GraalVM compiler in a
programming-language-agnostic way. To assess the quality of the generated data,
we trained several machine learning models to replace compiler heuristics for
loop-related optimizations. The trained models perform equally well to the
highly-tuned compiler heuristics when comparing the geometric means of
benchmark suite performances. Larger impacts on few single benchmarks range
from speedups of 20% to slowdowns of 17%. The presented approach can be
implemented in any dynamic compiler. We believe that it can help to analyze
compilation decisions and leverage the use of machine learning into dynamic
compilation.
