  The growing memory footprints of cloud and big data applications mean that
data center CPUs can spend significant time waiting for memory. An attractive
approach to improving performance in such centralized compute settings is to
employ prefetchers that are customized per application, where gains can be
easily scaled across thousands of machines. Helper thread prefetching is such a
technique but has yet to achieve wide adoption since it requires spare thread
contexts or special hardware/firmware support. In this paper, we propose an
inline software prefetching technique that overcomes these restrictions by
inserting the helper code into the main thread itself. Our approach is
complementary to and does not interfere with existing hardware prefetchers
since we target only delinquent irregular load instructions (those with no
constant or striding address patterns). For each chosen load instruction, we
generate and insert a customized software prefetcher extracted from and
mimicking the application's dataflow, all without access to the application
source code. For a set of irregular workloads that are memory-bound, we
demonstrate up to 2X single-thread performance improvement on recent high-end
hardware (Intel Skylake) and up to 83% speedup over a helper thread
implementation on the same hardware, due to the absence of thread spawning
overhead.
