Spark SQL -- Tungsten

Concept

Spark uses two engines to optimize and run the queries - Catalyst and Tungsten, in that order. Catalyst basically generates an optimized physical query plan from the logical query plan by applying a series of transformations like predicate pushdown, column pruning, and constant folding on the logical plan. This optimized query plan is then used by Tungsten to generate optimized code, that resembles hand written code, by making use of Whole-stage Codegen functionality introduced in Spark 2.0. This functionality has improved Spark’s efficiency by a huge margin from Spark 1.6, which used the traditional Volcano Iterator Model.

The goal of Project Tungsten is to improve Spark execution by optimising Spark jobs for CPU and memory efficiency (as opposed to network and disk I/O which are considered fast enough).

• Off-Heap Memory Management using binary in-memory data representation aka Tungsten row format and managing memory explicitly,
• Cache Locality which is about cache-aware computations with cache-aware layout for high cache hit rates
• Whole-Stage Code Generation (aka CodeGen).