What is it?
- Programming model for processing large amount of distributed dataset.
- The MapReduce model is based on the idea of dividing a large dataset into smaller chunks, which are then processed in parallel by multiple machines. The processing is done in two stages: the "map" stage and the "reduce" stage.
- You decide what KEY:VALUE pairs MAP will generate, and you decide how the data will be aggregated in REDUCE step.
How?
In the map stage, the input data is divided into smaller chunks, and a "map" function is applied to each chunk to transform it into a set of intermediate key-value pairs. These intermediate key-value pairs are then shuffled and sorted by key, so that all the values with the same key are grouped together.
In the reduce stage, a "reduce" function is applied to each group of values, which combines them in some way to produce a set of output values. The final output of the MapReduce computation is a set of key-value pairs.