Distributed computation · 2004 · Jeffrey Dean & Sanjay Ghemawat
MapReduce: Simplified Data Processing on Large Clusters
Restrict programs to map, shuffle, and reduce so the runtime can own partitioning, scheduling, data movement, replay, and straggler handling.
The central move
Restrict programs to map, shuffle, and reduce so the runtime can own partitioning, scheduling, data movement, replay, and straggler handling.
Why it had to exist
Large data jobs repeatedly reimplemented the same distributed execution and recovery machinery. Most application authors needed a safer boundary, not unrestricted distributed control.
Where it leads
Restricted data-parallel API → Hadoop → cluster dataflow systems and managed analytics.