  We present Stocator, a high performance object store connector for Apache
Spark, that takes advantage of object store semantics. Previous connectors have
assumed file system semantics, in particular, achieving fault tolerance and
allowing speculative execution by creating temporary files to avoid
interference between worker threads executing the same task and then renaming
these files. Rename is not a native object store operation; not only is it not
atomic, but it is implemented using a costly copy operation and a delete.
Instead our connector leverages the inherent atomicity of object creation, and
by avoiding the rename paradigm it greatly decreases the number of operations
on the object store as well as enabling a much simpler approach to dealing with
the eventually consistent semantics typical of object stores. We have
implemented Stocator and shared it in open source. Performance testing shows
that it is as much as 18 times faster for write intensive workloads and
performs as much as 30 times fewer operations on the object store than the
legacy Hadoop connectors, reducing costs both for the client and the object
storage service provider.
