The SOFA optimizer for UDF-heavy data flows

Recent years have seen an increased interest in large-scale analytical data flows on non-relational data. In many novel application areas the predominant building blocks of such data flows are user-defined predicates or functions (UDFs). However, the heavy use of UDFs is not well taken into account for data flow optimization in current systems. SOFA is a novel and extensible optimizer for UDF-heacy data flows. It builds on a concise set of properties for describing the semantics of Map/Reduce-style UDFs and a small set of rewrite templates, which use these properties to find a much larger number of semantically equivalent plan rewrites than possible with traditional techniques. A salient feature of SOFA is extensibility: We arrange user-defined operators and their properties into a subsumption hierarchy, which considerably eases integration and optimization of new operators. We evaluate Sofa on a selection of UDF-heavy data flows from different domains and compare its performance to three other algorithms for data flow optimization. Our experiments reveal that Sofa finds efficient plans, outperforming the best plans found by its competitors by a factor of up to six.

Screencast of Demonstration


Download of VM image


Publications

  • Rheinländer, A., Leser, U., Graefe, G.: Optimization of Complex Dataflows with User-Defined Functions. ACM Computing Surveys, vol. 50 (3), article 38, 39 pages, 2017.
  • Rheinländer, A., Heise, A., Hueske, F., Leser, U., Naumann, F.: SOFA: An Extensible Logical Optimizer for UDF-heavy Data Flows. Information Systems, Vol. 52(C), pp. 96-125, 2015.
  • Rheinländer, A., Beckmann, M., Kunkel, A., Heise, A., Stoltmann, T., Leser, U.: Versatile optimization of UDF-heavy data flows with Sofa. Demo, SIGMOD, 2014.
  • Rheinländer, A., Heise, A., Hueske, F., Leser, U., Naumann, F.: SOFA: An Extensible Logical Optimizer for UDF-heavy Dataflows. Technical Report, CoRR, arXiv:1311.6335, 2013.