  Distributed storage systems for large clusters typically use replication to
provide reliability. Recently, erasure codes have been used to reduce the large
storage overhead of three-replicated systems. Reed-Solomon codes are the
standard design choice and their high repair cost is often considered an
unavoidable price to pay for high storage efficiency and high reliability.
  This paper shows how to overcome this limitation. We present a novel family
of erasure codes that are efficiently repairable and offer higher reliability
compared to Reed-Solomon codes. We show analytically that our codes are optimal
on a recently identified tradeoff between locality and minimum distance.
  We implement our new codes in Hadoop HDFS and compare to a currently deployed
HDFS module that uses Reed-Solomon codes. Our modified HDFS implementation
shows a reduction of approximately 2x on the repair disk I/O and repair network
traffic. The disadvantage of the new coding scheme is that it requires 14% more
storage compared to Reed-Solomon codes, an overhead shown to be information
theoretically optimal to obtain locality. Because the new codes repair failures
faster, this provides higher reliability, which is orders of magnitude higher
compared to replication.
