
Building Colossus and erasure coding

Distributed databases with Peter Mattis
Topic 4 of 13
Published Sep 30, 2026 · Clip 17:27–24:01 (6:34)
Peter describes Colossus, the successor to GFS, a distributed file system with a flat namespace similar to S3 blob storage. He explains erasure coding with Reed-Solomon — nine chunks of which any five reconstruct the data — as a big breakthrough that reduced storage overhead from triplication to roughly 2x while increasing redundancy. He covers GFS's ~1000-machine scalability limit and the single GFS master bottleneck, solved by storing metadata in Bigtable, including the circular dependency hack where a foundational Bigtable didn't use Colossus.