  Web search and other large-scale web data analytics rely on processing
archives of web pages stored in a standardized and efficient format. Since its
introduction in 2008, the IIPC's Web ARCive (WARC) format has become the
standard format for this purpose. As a list of individually compressed records
of HTTP requests and responses, it allows for constant-time random access to
all kinds of web data via off-the-shelf open source parsers in many programming
languages, such as WARCIO, the de-facto standard for Python. When processing
web archives at the terabyte or petabyte scale, however, even small
inefficiencies in these tools add up quickly, resulting in hours, days, or even
weeks of wasted compute time. Reviewing the basic components of WARCIO and
analyzing its bottlenecks, we proceed to build FastWARC, a new high-performance
WARC processing library for Python, written in C++/Cython, which yields
performance improvements by factors of 1.6-8x.
