AI Infrastructure and GPU Efficiency

Twisting the rules of building software: Bending Spoons (the team behind Evernote)

Topic 3 of 8

Published Oct 23, 2024 · Clip 19:17–23:07 (3:50)

Federico and Luca describe Remini's AI inference load: over 2,000 inferences per second on average with peaks of 8,000, requiring around 4,000 GPUs dynamically allocated. They explain the dual optimization problem of deallocating unused GPUs to save money while cloud providers' booking models make GPU availability costly, solved with predictive algorithms. Luca ties this efficiency pressure to bootstrapping profitability: raising equity capital in Italy in 2014 was borderline impossible, so they grew via cash flow and bank debt, which bred resourcefulness.