Introducing Inspect Sentinel

We’re excited to announce Inspect Sentinel, a package for watching an agent while it runs and intervening when it does something it should not. A sentinel is configured on a task with Task(sentinel=...) and is made of two kinds of function:

• A monitor observes a step of the agent’s execution, such as a tool call it is about to make, and reports how suspicious it is. A monitor never acts.
• A protocol decides what happens at that step: proceed, reject, modify, terminate, or escalate. It may consult monitors first, or decide on its own. A protocol with no monitors is a rule.

Keeping the scoring separate from the deciding means every threshold lives in a protocol, where it can be calibrated, and every monitor can be run with nothing acting on it, to see what it would have flagged.

A Rule

The simplest sentinel is a rule. This one rejects any bash() command that reaches for the network:

from inspect_sentinel import (
    BeforeToolCall, Context, Decision, Escalations,
    Protocol, call_text, find_words, protocol,
)

@protocol
def no_network() -> Protocol:
    async def decide(
        context: Context,
        step: BeforeToolCall,
        escalations: Escalations,
    ) -> Decision | None:
        if step.call.function != "bash":
            return None  # abstain
        used = find_words(call_text(step.call), ["curl", "wget"])
        if used:
            return Decision.reject(
                f"command uses {', '.join(used)}",  # for the log
                message=f"`{used[0]}` is not allowed in this task.",  # for the agent
            )
        return Decision.proceed()

    return decide


@task
def network_rule() -> Task:
    return Task(
        dataset=[FIX_SCRIPT, FETCH_DATA, COUNT_ROWS],
        solver=react(tools=[bash(timeout=60), python(timeout=60)]),
        sandbox=("docker", "compose.yaml"),
        sentinel=no_network(),
    )

The type of step says when the function runs: BeforeToolCall runs before each tool call executes, and AfterToolCall, BeforeGenerate, and AfterGenerate cover the other points in the agent’s loop. A rejection carries two texts. The explanation goes to the log and the agent never sees it. The message is what the agent receives as the tool call’s error, so it can carry on.

Rules are cheap and run without a model call, but they match text and cannot prove a call safe. Anything that needs an understanding of shell or code goes to an LLM monitor.

An LLM Monitor

A monitor asks a model how likely each tool call is to break each part of a policy, scoring each dimension separately:

@monitor
def suspicion(model: str | None = None, role: str | None = None) -> Monitor:
    async def check(context: Context, step: BeforeToolCall) -> Observation | None:
        if step.call.function == "submit":
            return None
        return await context.host.score(
            step,
            DIMENSIONS,
            preprocessor=StepPreprocessor(input_window=LastTurns(3)),
            policy=POLICY,
            model=model,
            role=role,
        )

    return check

context.host.score() builds the prompt so that the task the agent was given is presented as trustworthy and the step is not, since everything in the step was written by or shown to the agent under observation and may try to talk the monitor out of flagging it. The monitor’s model calls are recorded apart from the agent’s and are not charged to the agent’s limits.

A monitor only reports, so to act on it wrap it in a protocol. threshold() turns scores into decisions, and a task has one sentinel but it can be several functions:

Task(
    ...,
    sentinel={
        "network": no_network(),
        "llm": threshold(suspicion(), reject_at=0.8),
    },
)

Run it with --model-role monitor=anthropic/claude-haiku-4-5 to give the monitor its own model. When a rejection comes from the threshold, the agent is told only that the call was not approved, so it learns nothing about what the monitor saw.

Composition

A mapping runs its functions under concurrent(): everything runs at once and the strongest decision wins, with terminate over reject over modify over escalate over proceed. sequential() runs its children in order, and the first that decides anything but escalate decides for the chain. That is how to put a cheap rule ahead of an expensive judge, and a person at the end who is asked only what the rules could not settle:

sentinel = sequential(
    [
        threshold([proposed_action()], escalate_at=0.6, terminate_at=0.9),
        human(stages=["tool_call"]),
    ]
)

human() asks through Inspect’s approval surfaces and records the person’s reason as the decision’s explanation. Later steps of the same agent can see those answers, so a monitor can mark the calls a person already reviewed rather than flag them again. An escalation that nothing handles fails closed and ends the sample, and handle_escalation("terminate") or handle_escalation("proceed") states a fixed policy instead.

In the Log Viewer

Each tool call a sentinel checked carries its checks inside the call’s card in Inspect View: a summary row naming the verdict and its explanation, an expandable tree of every check by path, and, for a rejected call, the message the agent received in place of the tool’s output. The monitor’s own model calls expand below, and the monitor role’s token usage is reported separately from the agent’s.

Replay with Inspect Scout

A monitor is a classifier, and you want to measure it before anything acts on it. as_scanner() turns a monitor or protocol into a Scout scanner that runs the same code over steps rebuilt from transcripts you already have:

from inspect_scout import scan, transcripts_from
from inspect_sentinel.scout import as_scanner

scan(
    scanners={"suspicion": as_scanner(suspicion())},
    transcripts=transcripts_from("./logs"),
    model_roles={"monitor": "openai/gpt-4o-mini"},
)

observe_only() records a monitor’s scores in an eval with nothing acting on them, which is how to collect a benign distribution before choosing a threshold. Monitors are portable by default, so the same monitor can later run outside an eval in a network proxy in front of a model API, which is planned.

Inspect Sentinel is under active development and its API may change. It requires inspect-ai 0.3.278 or later. The design documents are in the repository’s design directory, and the documentation walks through the examples above.