We’re excited to announce Inspect Steward, a system for automated execution and supervision of long-running evaluations. Steward plays the role of the person sitting in front of a running eval: it watches progress as a run unfolds, diagnoses problems as they surface, and adjusts runtime state in response. You launch a large run and walk away, and the questions that genuinely need you come back to you.
Large eval sets now routinely run for days. Over that span, providers go down, sandboxes time out, samples hang inside a tool call, and someone has to notice, decide what to do, and keep the run converging. Steward divides that work across three layers:
- A scheduled
steward tendloop runs every ten minutes with nobody watching. It respawns failed tasks, tunes concurrency against sandbox capacity and model throughput, and records what it observed. It exercises no judgement. - A coding agent supplies judgement. It diagnoses errors and groups them into classes, proposes what to do about each, handles stuck samples under standing grants, and writes up the run when it completes.
- You supply authority: rulings on error classes, tool call approvals, and final signoff.
To get started, install Steward and create a workspace:
pip install inspect-steward
mkdir -p ~/runs/swe-evals && cd ~/runs/swe-evals
steward initThis scaffolds an evalset.py, a _steward.yaml for standing rules, and an AGENTS.md that teaches your coding agent how to run the evaluation. Any script that ends in a call to eval_set() works as a definition, as does a config.py from Inspect Flow or a hawk.yaml from Hawk:
from inspect_ai import eval_set
from inspect_harbor import cais_swebenchpro, datacurve_deep_swe
eval_set(
tasks=[cais_swebenchpro(), datacurve_deep_swe()],
model=["openai/gpt-5.6-sol", "anthropic/claude-opus-5"],
)Then launch your coding agent from the workspace and ask it to run the evaluation. The agent starts with a short smoke run to confirm that configuration, infrastructure, and model access all work, launches the tasks, arms the tend timer, and works the decisions the run raises. The run depends on neither you nor the agent staying connected. When you come back, ask how it’s going, and the agent reports per-task progress, what it resolved on its own while you were gone, and anything waiting on you:
89 samples in cais_swebenchpro failed with the same
sandbox timeout, all inside a 30-minute window last
night. Automatic retry has already spent its attempts.
re-run them, score them as failed, or exclude them
from scoring?Errors Become Decisions
Inspect’s default behavior is to fail fast so you can see and fix a problem. Steward runs unattended for hours at a time, so it does the opposite: errors never halt a run, retries are automatic, and anything a retry cannot fix becomes a decision that waits in a queue. Samples that failed the same way are grouped into a class, so a provider outage is one question rather than two hundred. You answer with one of six dispositions (rerun, exclude, zero, score, accept, or dismiss), and when you find yourself answering the same question twice, the agent proposes recording it in _steward.yaml so it stops being asked:
preauthorized:
'error:ReadTimeout@*': rerun
policies:
- If more than 10% of the samples in a task error out, pause the run and
notify me right away.Steward also runs a scoring integrity scanner over every sample as it completes, flagging transcripts where the model read the grader, refused the task while producing something that looks like an answer, or retrieved remote data from a sandbox that should have had no route to the internet. Findings are decided the same way errors are. Nothing is dropped silently: every error is either retried or turned into a decision, and steward signoff refuses while any decision is unanswered.
Configure a notification channel (Slack, email, and many others via Apprise) and Steward will tell you when something needs a decision, when a run finishes, and when it cannot tend at all.
Get started with the Inspect Steward documentation.