  In parallel with big data processing and analysis dominating the usage of
distributed and cloud infrastructures, the demand for distributed metadata
access and transfer has increased. In many application domains, the volume of
data generated exceeds petabytes, while the corresponding metadata amounts to
terabytes or even more. This paper proposes a novel solution for efficient and
scalable metadata access for distributed applications across wide-area
networks, dubbed SMURF. Our solution combines novel pipelining and concurrent
transfer mechanisms with reliability, provides distributed continuum caching
and prefetching strategies to sidestep fetching latency, and achieves scalable
and high-performance metadata fetch/prefetch services in the cloud. We also
study the phenomenon of semantic locality in real trace logs, which is not well
utilized in metadata access prediction. We implement a novel prefetch predictor
based on this observation and compare it with three existing state-of-the-art
prefetch schemes on Yahoo! Hadoop audit traces. By effectively caching and
prefetching metadata based on the access patterns, our continuum caching and
prefetching mechanism significantly improves local cache hit rate and reduces
the average fetching latency. We replayed approximately 20 Million metadata
access operations from real audit traces, in which our system achieved 90%
accuracy during prefetch prediction and reduced the average fetch latency by
50% compared to the state-of-the-art mechanisms.
