Defining What Good Looks Likes - All Things Product Podcast with Teresa Torres & Petra Wille

Listen to this episode on: Spotify | Apple Podcasts

AI evals feel like a brand-new skill — but the hardest part is a question product teams have always had to answer: what does "good" actually look like?

Petra Wille and Teresa Torres dig into Teresa's recent article on AI evals and the one idea in it that hasn't changed at all. Teresa argues that measuring the quality of AI outputs is really the same work as defining outcomes, writing a PRD, or instrumenting a product — and that teams keep repeating an old mistake: measuring technical performance (uptime, speed, "was it concise?") instead of whether they actually created value for the customer. Drawing on the evolution of engagement metrics, a DocuSign example, and why a student and a professor need opposite definitions of "concise," they make the case for defining "good" in your own context — and never outsourcing it to off-the-shelf evals.

A grounding listen for any team worried they're falling behind on AI: much of the work is the work you already know how to do.

Key Takeaways

  • Defining "what good looks like" isn't new — AI evals are just the latest place that old work shows up.

  • Don't stop at technical performance. An eval suite should measure technology performance, customer value, and business value.

  • Don't outsource your definition of "good" to generic, off-the-shelf evals — define it in your own context.

  • "Good" is context-dependent: a concise answer is right for an expert and wrong for a beginner.

  • Metrics mature the same way every time — from universal (MAU) toward your product's real value-creation moment (e.g., DocuSign getting you to send a document).

  • AI amplifies whatever shape you're already in — automating a poorly-defined product or process just produces more of what doesn't work.

  • Underneath it all is one habit: be explicit about what you expect to happen, so you can test the assumption when you're wrong.

Notable Moments

  • "Good is: the customer did something that created value for the customer." — Teresa

  • "Don't outsource what good looks like. Really think about, in your context, what does good look like?" — Teresa

  • "Helpful is my new NPS." — Petra

  • "Even with conciseness, the definition of what good looks like changes with the context." — Teresa

  • "The habit that underlies metrics and outcomes is really just: what do you expect to happen?" — Teresa

Resources & Links:

Mentioned in this episode:

Petra Wille