  Recent years have seen an increased interest in large-scale analytical
dataflows on non-relational data. These dataflows are compiled into execution
graphs scheduled on large compute clusters. In many novel application areas the
predominant building blocks of such dataflows are user-defined predicates or
functions (UDFs). However, the heavy use of UDFs is not well taken into account
for dataflow optimization in current systems.
  SOFA is a novel and extensible optimizer for UDF-heavy dataflows. It builds
on a concise set of properties for describing the semantics of Map/Reduce-style
UDFs and a small set of rewrite rules, which use these properties to find a
much larger number of semantically equivalent plan rewrites than possible with
traditional techniques. A salient feature of our approach is extensibility: We
arrange user-defined operators and their properties into a subsumption
hierarchy, which considerably eases integration and optimization of new
operators. We evaluate SOFA on a selection of UDF-heavy dataflows from
different domains and compare its performance to three other algorithms for
dataflow optimization. Our experiments reveal that SOFA finds efficient plans,
outperforming the best plans found by its competitors by a factor of up to 6.
