  We propose repair pipelining, a technique that speeds up the repair
performance in general erasure-coded storage. By carefully scheduling the
repair of failed data in small-size units across storage nodes in a pipelined
manner, repair pipelining reduces the single-block repair time to approximately
the same as the normal read time for a single block in homogeneous
environments. We further design different extensions of repair pipelining
algorithms for heterogeneous environments and multi-block repair operations. We
implement a repair pipelining prototype, called ECPipe, and integrate it as a
middleware system into two versions of Hadoop Distributed File System (HDFS)
(namely HDFS-RAID and HDFS-3) as well as Quantcast File System (QFS).
Experiments on a local testbed and Amazon EC2 show that repair pipelining
significantly improves the performance of degraded reads and full-node recovery
over existing repair techniques.
