Spark Catalyst Internals

Spark catalyst is one of the secret sauce of Spark’s Operations on the structured data. Let’s take a deep look into its internals.

TreeNode Abstraction

TreeNode is the fundamental data type abstraction for the catalyst internals. This abstraction brings methods (such as foreach, map, flatmap, collect etc.) that helps to manipulate scala functional for manipulating the internal tree node structure. The contract of the tree node abstraction to the operators/ class that extending it is to define the list of children.

TreeNode library


Evaluate-able TreeNode


Query-able TreeNode


Transform-able TreeNode

Low-level RDD TreeNode