Optimism on LLM scaling and post-training

How Perplexity Works 🤖 — with Denis Yarats

Topic 4 of 12

Published May 30, 2024 · Clip 15:37–17:04 (1:27)

Denis expresses optimism that there's still a lot of room for model improvement. He cites GPT-4, which finished training almost two years before its release, as evidence that post-training techniques, better data, and synthetic data deliver major gains on top of a base model. Both pre-training and post-training matter, and he believes progress is far from saturating.