
Measuring answer accuracy with evals

How Perplexity Works 🤖 — with Denis Yarats
Topic 11 of 12
Published May 30, 2024 · Clip 40:02–44:12 (4:10)
Denis explains that measuring answer quality is a very hard problem requiring all available signals: implicit and explicit user signals, human evals with trained evaluators following clear guidelines, and LLM-as-judge automation for specific aspects like citation accuracy. The goal is to minimize human involvement, using humans only as the final judge on pre-processed information. The host notes this LLM-automated QA with humans in the loop could be a product in itself, and Denis confirms it's already Perplexity's bread and butter.