Benchmark replays
protect-mcp on two public agent benchmarks
AgentDojo and tau2-bench publish the runs behind their results. We passed every recorded tool call in those runs, in order, through the Cedar policy gate of protect-mcp 0.32.2, under rules fixed before each replay; no model was called, and each benchmark’s own labels and data are the answer key.
- Not an independent evaluation. ScopeBlind makes protect-mcp, the gate under test, and wrote the policies and adapters it was tested with.
- The designs, replays, scorers and write-up were produced by ScopeBlind with AI coding agents (Claude, by Anthropic).
AgentDojo
36,679 published run files from 29 model and defence pipelines in 4 suites. The attack set is the 5,570 attack traces in which the agent made at least one call toward the attacker’s goal; the honest set is the 1,588 honest runs AgentDojo scored as successful.
| Policy | First attacker call denied | Successful honest runs with no denial |
|---|---|---|
| Part A: destinations only | 76.6%4,266 of 5,570; 75.5 to 77.7 | 97.5%1,548 of 1,588; 96.6 to 98.1 |
| Full policy: parts A, B and C | 98.7%5,496 of 5,570; 98.3 to 98.9 | 97.2%1,543 of 1,588; 96.2 to 97.9 |
Part A says one thing: money, e-mail, invitations, web posts and link fetches may go only to a destination already on file in the owner’s uninjected data, or to one the user named in the request. Its lists come from the uninjected environments, extracted by a script that reads no attack definition.
The part B and C rules were written after reading AgentDojo’s injection tasks. The full policy’s numbers therefore show what an owner who knew these attack classes could write, not what a policy written blind would catch. Part A was drafted at the same time, by the same authors, with the same knowledge; what keeps it closer to attack-agnostic is that its rule is generic and its lists are the owners’ own records.
tau2-bench retail
1,824 published retail runs (4 models, 114 tasks, 4 trials), replayed through 20 Cedar rules compiled from the shop’s written policy, with 52 planted failures and 35 controls. The answer key is the final database state, compared by hash with the gold state, and the pass conditions were fixed in a registration signed before the first replay.
- C0, C1, C3 and C5 pass: the database-only label of every published run reproduces, 52 of 52 planted failures are caught by the predicted layer for the predicted reason, every receipt chain verifies offline, and a second pass gives identical decisions.
- C2 passes as written: no write was refused in a passing run or a control without a mechanical ground. Under a stricter reading, in which any refused write in a passing run fails, it would fail: 170 refused writes in 149 of the 1,358 passing runs.
- C4 is descriptive, with no threshold.
Under enforcement, 33 of the 1,358 runs that the answer key accepts would not have reached the gold database state as recorded.
What this does not show
- What happens after a denial. The replay is counterfactual from the first refused call: the recorded run goes on as if the call had run, so nothing here says whether a live agent would retry, give up or find another route.
- Attacks these benchmarks do not contain. Only AgentDojo’s recorded attacks were replayed. An attacker who knows the policy can pick a destination that is on file, an action of the kind the user asked for, or a channel the policy leaves open.
- Anything a tool-boundary policy cannot see. The gate sees tool calls, not answers, so output-only manipulation is out of reach, as are reads, leaks through allowed channels and wrong values inside allowed bounds. protect-mcp does not stop prompt injection as such; it can only refuse specific calls.
- A live deployment. No model ran. The tau2 customer is a simulated user following a script, and the shop’s database is synthetic.
- That the policies are good policies. They were written by the gate’s authors, with the benchmarks in view.
The evidence
The policies, scripts, results, signed samples and the caveats in full are in the repository, with the commands to reproduce the replays and to verify the receipts offline.