The Hidden Signal in Production AI Logs

Source: original

Free Lesson

The Hidden Signal in Production AI Logs

Part of The AI Evaluation Handbook

•

Hosted by Jason Liu and Scott Clark

173 students

Copy link

173 students

Copy link

In this video

Collapse

00:00:00Introduction to AI Observability and the Data Flywheel00:02:12Speaker Introduction: Scott Clark's Background in AI Tools00:05:36Defining AI Analytics vs. Traditional Observability00:09:07Product Demo: Analyzing an AI Agent with Distributional

00:16:04How It Works: The Unsupervised Data Flywheel for AI Analytics00:22:03The Three Pillars: Enrichment, Analysis, and Publishing Insights00:26:19Getting Started and Contrasting Analytics with Offline Evaluations00:28:09The Limitations of Offline Evals and Discovering "Unknown Unknowns"00:32:32Identifying When to Implement Advanced AI Analytics00:36:13Common AI Failure Modes in Production Environments00:39:33Motivating Leadership to Invest in AI Observability00:42:27Final Thoughts: The Evolution and Future of AI Analytics

View all

What you'll learn

Enrich AI logs with behavioral metrics

Learn to augment production logs with statistical metrics, evals, and LLM-as-judge signals

Apply clustering and anomaly detection

Use high-dimensional clustering and drift detection to surface hidden behavioral patterns

Build actionable analysis workflows

Create systematic processes to translate behavioral signals into product improvements

Why this topic matters

Production AI fails when teams can't see what's actually happening. This lesson teaches you to decode the black box—transforming messy logs into behavioral insights that reveal why your AI works or breaks. You'll gain the analytical framework to systematically improve AI products, moving beyond surface metrics to understand and fix real user issues at scale.

You'll learn from

Jason Liu

Consultant at the intersection of Information Retrieval and AI

Jason has built search and recommendation systems for the past 6 years. He has consulted and advised a dozens startups in the last year to improve their RAG systems. He is the creator of the Instructor Python library.

Scott Clark

Co-Founder & CEO, Distributional

Scott Clark is Co-Founder and CEO of Distributional, the first enterprise platform that analyzes hidden behavioral signals from production log data to help you continuously improve your AI products.

The platform uses unsupervised learning to surface subsets of log data corresponding to shifts, clusters, or outliers in AI behavior that a user can investigate and track through custom filters, metrics, and alerts to catch issues and adapt this analysis over time. This empowers AI teams to better understand the behavior of their users and AI applications so that they can fix and improve those applications with confidence.

Scott was previously the Co-Founder and CEO of SigOpt, an enterprise AI optimization platform. After selling SigOpt to Intel in 2020, Scott was VP of AI and HPC engineering within the Supercomputing organization. Prior to SigOpt, Scott worked and performed research at various technology companies, universities, and national labs around the world. Scott holds a PhD in Applied Math and MS in Computer Science from Cornell University and BS degrees in Mathematics, Physics, and Computational Physics from Oregon State University. Scott was named to the Forbes 30 under 30 for enterprise tech in 2016.

worked with

See all products from Applied LLMs

Share this lesson

173 students

Copy link

Share this lesson

173 students

Copy link

Go deeper with a course

Featured in Lenny’s List

Systematically Improving RAG Applications