
Data Locality and Memory

#60 - Software Tradeoffs and How to Make Good Programming Decisions - Tomasz Lelek
Topic 7 of 10
Published Oct 18, 2021 · Clip 25:06–33:02 (7:57)
Tomasz explains data locality: moving computations to the data rather than data to computations, as done in Spark and Hadoop. Computations are serialized and sent to data nodes where executors apply them locally, with results coalesced afterward. He covers join operations, network shuffling as a key optimization target, and choosing partitioning schemes aligned with query patterns. He notes this applies to microservices too—push computation to services holding the data and return only results—and discusses how modern frameworks revive the stored-procedure idea with better abstraction and testability.