  We develop a tool called PipeGen for efficient data transfer between database
management systems (DBMSs). PipeGen targets data analytics workloads on
shared-nothing engines. It supports scenarios where users seek to perform
different parts of an analysis in different DBMSs or want to combine and
analyze data stored in different systems. The systems may be colocated in the
same cluster or may be in different clusters. To achieve high performance,
PipeGen leverages the ability of all DBMSs to export, possibly in parallel,
data into a common data format, such as CSV or JSON. It automatically extends
these import and export functions with efficient binary data transfer
capabilities that avoid materializing the transmitted data on the file system.
We implement a prototype of PipeGen and evaluate it by automatically generating
data pipes between five different DBMSs. Our experiments show that PipeGen
delivers speedups up to 3.8x compared with manually exporting and importing
data across systems using CSV.
