  Some important problems, such as semantic graph analysis, require large-scale
irregular applications composed of many coordinating tasks that operate on a
shared data set so big it has to be stored on many physical devices. In these
cases, it may be more efficient to dynamically choose where code runs as the
applications progresses. Many programming environments provide task migration
or remote function calls, but they have sharp trade-offs between flexible
composition, portability, performance, and code complexity.
  We developed Two-Chains, a high performance framework inspired by active
message communication semantics. We use the GNU Binutils, the ELF binary
format, and the RDMA network protocol to provide ultra-low granularity
distributed function composition at runtime in user space at HPC performance
levels using C libraries. Our framework allows the direct injection of function
binaries and data to a remote machine cache using the RDMA network. It
interoperates seamlessly with existing C libraries using standard dynamic
linking and load symbol resolution. We analyze function delivery and execution
on cache stashing-enabled hardware and show that stashing decreases latency,
increases message rates, and improves noise tolerance. This demonstrates one
way this method is suited to increasingly network-oriented hardware
architectures.
