We’re excited to announce Inspect Scout, a tool for in-depth analysis of AI agent transcripts. With Scout, you can easily:
- Detect issues like misconfigured environments, refusals, and evaluation awareness using LLM-based or pattern-based scanners.
- Analyze transcripts from Inspect, Arize Phoenix, LangSmith, Logfire, MLFLow, W&B Weave, Claude Code, or custom sources via the capture and import APIs.
- Develop scanners interactively, exploring transcripts and scan results visually in Scout View.
- Validate scanner accuracy against human-labeled examples.
- Handle complex scanning requirements like multi-agent transcripts, compaction, and context-window chunking.
- Scale to thousands of transcripts with parallel processing, batching, and fault tolerance.
Scout also includes a validation framework for measuring scanner accuracy against human-labeled examples, so you can iteratively refine your scanners with confidence.
We’re especially appreciative of the feedback we received from UK AISI, US CAISI, METR, Apollo, and many others during Scout’s development. Their paper on Seven Simple Steps for Log Analysis in AI Systems goes in depth on best practices for transcript analysis including many practical examples.
Get started with the Inspect Scout documentation.