This week the UK AI Security Institute released Transect, an open source package for analyzing transcripts from long-horizon agent evaluations. Transect is built on Inspect Scout, and we worked closely with AISI on its development. Their blog post and the accompanying paper have the full story. Here are the highlights.
As agents take on tasks that run for days and delegate to sub-agents, a final score says little about what the agent tried, where it got stuck, or how the setup shaped its work. The transcript holds that detail, but a single run can reach millions of tokens (the AI R&D run analyzed in the paper generated nearly 13 million), well beyond what a reviewer can read in full. Pointing an LLM judge at the transcript only moves the problem, since judged labels cannot be taken at face value either. Transect is built around that fact: every classification it produces carries provenance and reliability information, so reviewers can see how a label was produced and where judges disagreed.
You supply a transcript, the task context, and the activity categories you want to distinguish. The spec is a YAML file that is reusable across runs of the same evaluation:
context: >
An open-ended AI R&D task: the agent is given a research brief
and has to implement the described method, run experiments with
it, and write up its findings. It can delegate to sub-agents.
phases:
- label: planning
description: Understanding the brief and choosing an approach.
- label: implementation
description: Writing and debugging the method and experiment code.
- label: experimentation
description: Running experiments and analysing their results.
- label: writeup
description: Writing the findings up into the report.One call runs the pipeline and opens the report with a Scout viewer wired in for deep links into the transcript:
from transect import transect
results = transect(
"logs/my_run.eval",
"spec.yaml",
judge_models="anthropic/claude-sonnet-4-6",
k_rolls=3,
verify=True,
)The report puts everything on one turn-based timeline: the phase band with a per-turn judge agreement strip, human interventions, token telemetry with context compactions marked, a swimlane for each sub-agent with its classified role, and token spend broken down by phase or sub-agent. Selecting a phase shows its narration, excerpts, and reliability, and links back to the transcript.
Three things about the design stand out:
- Provenance over confidence. Labels link to the turns they describe and record how they were made. With repeated judgements or a cohort of judge models, the report shows where they disagree. Agreement does not prove a label is right, but disagreement tells you where to look. Judgements are saved, so an analysis can be reopened without new model calls.
- Structural first. Token use, compactions, human interventions, and sub-agent spans are extracted directly from the log with no model calls. A structural run is free and is the right place to draft your phase and sub-agent vocabulary before spending on judges.
- Frames. Everything the scanners extract is exposed as pandas dataframes, so custom analyses and custom report layers build on the same data. The AISI post shows one such analysis, tracing four documents (a research plan, baseline code, an experiments draft, and a blind self-review) as they are written and read across sub-agents over a six-day run.
Transect reads Inspect .eval logs and OpenClaw telemetry exports. It also recognizes Petri audits, placing the auditor’s turns on the axis with each target trajectory as its own lane. The package ships skills for Claude Code and similar coding agents that walk through drafting a spec and iterating on label reliability.
AISI’s post also makes the case for why this matters now. Systematic transcript review was part of how they identified unsanctioned agent actions in a security incident earlier this year, and their work on cheating in evaluations shows why apparent successes need checking for how the agent got there. Tools that preserve evidence and the record of how an analysis was produced are what make oversight at this scale possible.
Transect is available on GitHub. Read the AISI announcement and the paper for more.