High Order Functions, Roots of MapReduce

Hadoop has its roots in functional programming, which is exemplified in languages such as Lisp and ML. A key feature of functional programming language is the concept of higher-order functions, or functions that can accept another functions as arguments. Two common built-in order functions are map and fold, illustrated in below figure.

Given a list, map takes as an argument a function f (that takes a single argument) and applies it to all elements in a list. Given a list, fold takes as arguments a function g (that takes two arguments) and an initial value: g is first applied to the initial value and the first item in the list, the result of which is stored in an intermediate variable. This intermediate variable and the next item in the list serve as the arguments to a second application of g, the results of which are stored in the intermediate variable. This process repeats until all items in the list have been consumed; fold then returns the final value of the intermediate variable. Typically map and fold are used in combination. 

 

We can view map as a consise way to represent the transformation of a dataset as defined by the function f. In the same vein, we can view fold as an aggregation operation, as defined by the function g. On immediate observation is that the application of f to each item in a list (or more generally, to elements in a large dataset) can be parallelized in a straightforward manner, since each functional application in isolation. In a cluster, these operations can be distributed across many different machines. The fold operation, on the other hand, has more restrictions on data locality. Elements in the list must be "brought together" before the function g can be applied. However, many real-world applications do not require g to be applied to all elements of the list. To the extent that elements in the list can be devided into groups, the fold aggregations can also proceed in parallel. Furthermore, for operations that are commutative and associative, significant efficiencies can be gained in the fold operation through local aggregation and appropriate reordering.

 

In a nutshell, we have described MapReduce. The map phase in MapReduce roughly corresponds to the map operation in functional programming, whereas the reduce phase roughly corresponds to the fold operation in functional programming.

 

There is a interesting blog post on how to implement high order functions(HOF) in Java.

TODO --> continued with concrete example, maybe in Java, to illustrate the concept of HOF.