Proposal: standardize operator-friendly tool failure metadata for common MCP cases #2930
Replies: 12 comments 3 replies
|
I like the direction of keeping this lightweight rather than introducing a new error framework one thing I would suggest is separating machine-readable classification from the human-readable error message. For example, a tool could return a stable error code like:
alongside a descriptive message and optional metadata (retry delay, required capability, login URL, etc.) that would let clients implement consistent behavior (retry, prompt for authentication, disable UI actions, etc.) without relying on parsing free-form text while still giving operators useful debugging information. I also think starting as a best-practice recommendation is the right approach. If multiple MCP servers converge on the same conventions it would provide real implementation experience before considering a formal SEP. one additional category that may be worth considering is something like overall having a small recommended set of standardized classifications seems like it would improve interoperability without adding much complexity |
|
Agree on keeping this best-practice-first and separating machine classification from the human message. One thing I'd add: the codes in @Mohataseem89's list actually span three groups that call for different client behavior, and a single flat list hides that. It may help to carry a small
I'd argue governance is the one worth making explicit, because it's the current gap. It isn't the same as Concretely: keep the granular |
|
Ganerate Proposal based [2025 05 12 Green Paper on Establishing a Somaliland Bar Association Law.docx](https://github.com/user-attachments/files/29873698/2025.05.12. into: advocacy grants: |
|
Ganerate proposal |
|
On your question about existing implementations: yes. I've been building this as a transparent proxy called mcp-fuse, and the metadata it converged on overlaps heavily with this thread. Agree with @Mohataseem89 on separating machine classification from the human message, and with @HarperZ9 that the codes imply different client behaviors. In my schema that ended up as three fields doing distinct jobs: category: a small closed enum (transient, rate_limit, timeout, auth, permission, invalid_input, not_found, resource_exhausted, permanent, unknown). Roughly your list plus the retry-relevant cases. retryable plus an optional retry directive (strategy, "not before" timing, max attempts). This is where this thread meets #3188, which is asking about the timing half of the same problem; the two probably belong together. agentGuidance: one short imperative sentence intended for the model, capped in length, never a stack trace. Raw diagnostics go in a separate detail field that clients shouldn't forward to the model at all. In agent contexts this split matters as much as the codes, because the error text itself is what burns context. One implementation lesson worth folding into the convention: whether a client may auto-retry has to depend on the tool's declared idempotency (readOnlyHint / idempotentHint), not only on the error class. An ambiguous failure on a tool with side effects should never be silently replayed. A nice side effect of making that normative is that servers finally get a real incentive to annotate their tools correctly. Schema draft with JSON Schema and examples: https://github.com/YoadElkayam/mcp-fuse/tree/main/spec. It runs in the wild as a proxy, so there's measured data as well (benchmarks show it keeping roughly 96 to 98 percent of error payload tokens out of model context). Happy to reshape any of it toward whatever convention this group lands on. Starting as documentation guidance before an SEP sounds right to me. |
|
@YoadElkayam Happy to close this one as a duplicate if the maintainers prefer a single venue. Either way I'd rather help ship one proposal than seed a third convention. I can also contribute the enterprise operator perspective on what a support-facing failure signal has to include to actually shorten a ticket, which is the requirement I'm closest to. One thing I want to flag as at risk. The thread has developed three consumers of this payload: the model reads agentGuidance, the client switches on category and retryability, and the operator reads the granular code. The first two have gotten most of the attention here, and the third is the one that motivated my original post. Faster troubleshooting needs one more field that the other two consumers don't care about: a server-generated correlation id that a user can paste into a support ticket and an operator can grep for in server logs. Today that reconciliation is done by matching timestamps, which fails exactly when you need it most, on a busy multi-tenant server. It costs one optional string and it's the difference between "a tool call failed sometime around 10:04" and a specific request. I'd put it in the same payload rather than leaving it to convention, since a field only operators use is precisely the field that never gets adopted if it's merely suggested. On next steps, I'd like to join the effort @YoadElkayam and @aurumflux20 described in #3188 rather than run this thread in parallel. Concretely, I'd take two pieces: the classification vocabulary, meaning the code set, the code-to-class mapping, and the governance case with its client-behavior rules; and the docs-first guidance page that lands before the SEP. |
|
@johnyzaguirre-glean quick status from the shared draft: aurumflux20's sections (evidence + effect declaration) are merged, the conformance battery is running, and the two remaining gaps are yours: the category vocabulary with the code-to-class mapping, and the correlationId text. No rush if it's in progress, just flagging that your pieces are now the critical path before we take it to the Interceptors WG. If your plate is full, say so and we'll draft from this thread's earlier comments and credit you; happy either way. Draft: https://github.com/YoadElkayam/mcp-fuse/tree/main/sep (drafted with Claude Code, disclosure as in #3188) |
|
A field result that bears on §4.3 and, I think, on something this proposal does not yet cover. I read eight agent-payment toolkits published by card processors and billing platforms — MCP servers and agent SDKs — for one question: when a payment succeeds and the reply does not arrive, does a retry charge the customer again. Five of the eight can. Write-up with method and limits: https://aurumflux.co/agent-toolkits/ The finding I did not expect: In three of them the idempotency mechanism is implemented correctly, and the tool description defeats it. Two require an idempotency key from the caller and never generate one server-side — the right design. Their parameter description then says the value must be "unique for every request." To a human integrator that is correct: "request" means the purchase, and a retry is the same purchase. But an MCP server's only caller is a model, and the description is the specification it receives. It reads "unique for every request" as every call I make, mints a fresh key on retry, and the platform correctly records a second payment. In one case the platform ships the exact recovery endpoint for an unknown outcome, whose precondition is holding the original key — the key the description told the caller to change. A third scopes the platform's idempotency header per client rather than per call, with no per-call path and nothing ever assigning it. Unset: no protection. Set, as its configuration API invites: one value on every request, so distinct purchases collide and the second silently returns the first one's result. Why this is an MCP-shaped problem rather than a vendor one. In a normal SDK, documentation is read once by a human who forms a mental model and writes code that persists. In MCP the text is re-read on every call by a caller with no memory of the last one, and it is executed rather than interpreted. That inverts what good documentation looks like. "Unique identifier for this request" is careful, accurate prose and an active defect. None of these teams wrote bad code; they wrote docs for the wrong reader. Where it touches this proposal. §4.3's rule — that an outcome which could not be determined is never reported as one that did not happen — governs what a server says. This is the mirror: what a server asks for. A tool can be perfectly conformant on its failure metadata and still guarantee duplicate side effects through its parameter descriptions. If the SEP is going to define operator-facing metadata, it may be worth a line on retry-stability of caller-supplied identifiers: that where a tool accepts an idempotency key, its description must state the value is stable across retries of the same logical operation and distinct between different ones. Both halves, because in the per-client case the two requirements cannot currently be satisfied at the same time. I am not proposing text for it here — this thread already has enough in flight, and @YoadElkayam's draft has the two open items to close first. Raising it because "the description is the interface" is a class of defect I had not seen named, and it is cheap to prevent at spec level and expensive to find afterwards. Limits. All eight are code reads; I did not run agents against them and count duplicate charges, so this is a claim about what the instruction says rather than a measured rate. Eight is not a random sample. One further toolkit could not be assessed — its descriptions are served from a credentialed endpoint, I tried unauthenticated introspection, got a 401, and stopped rather than authenticate into production, so it is recorded unassessable rather than safe. Every one of the five was told before publication, none is named while its finding is open, and I asked each whether their backend deduplicates independently — where it does, the finding reduces to a documentation gap and I will publish that correction as prominently as the claim. The three that were clean are named with credit on the page. One of them documents idempotency to the model explicitly — "handlers may run multiple times for the same event", with upsert keyed on the platform identifier — which is the best example I have of writing tool documentation for a model caller on purpose, and more useful to this group than any of the findings. |
|
This looks like a declaration-side companion to §4.3’s reconciliation-verdict rule, not another error category. For caller-supplied idempotency keys, the replay gate only works if the logical operation keeps one stable identity while transport attempts change underneath it. A compact normative rule could be:
I would also put this in the tool contract, not only the error payload: by the time §4.3 handles an ambiguous result, rotating the key has already destroyed the reconciliation identity. A conformance case is cheap: apply the effect, drop the response, retry, then assert one effect and the same key across attempts. A second case should make a distinct operation and assert a distinct key, so a client cannot pass by reusing one process-wide value. |
|
Folded in. It is 4.4 in the draft now, "caller-supplied identifiers", with @HarperZ9's wording as the normative text and the toolkit reading as motivation, both credited. Kept it in the tool contract rather than the payload for exactly the reason given: by the time 4.3 is handling an ambiguous result, a rotated key has already destroyed the identity reconciliation depends on. Also added both conformance cases to the battery's open items; the distinctness one matters, a client should not be able to pass by reusing one process-wide value. "The description is the interface" also went into the draft as a named idea. It explains more than this one finding and I suspect it earns its own guidance page someday. https://github.com/YoadElkayam/mcp-fuse/tree/main/sep (drafted with Claude Code, same disclosure as above) |
|
Seller/operator note from monetized MCP / x402-style tools that may belong in the category vocabulary (or as a payment-adjacent subclass), separate from the idempotency / §4.3–4.4 work already in flight. We’ve seen three failure shapes that look identical to the client (“no demand” / “wallet problem” / “tool returned data”) until an operator has a machine-readable class:
These are common enough on paid tool surfaces that a docs-first convention (machine class + human message + correlationId, per earlier comments) would shorten tickets the same way |
|
Welcome, and these are exactly the right shape of report: three failures that are invisible until someone gives them a name. Folded in, with one design boundary kept: the category enum stays closed, because it is a client behavior contract and "payment header invalid" demands the same client behavior as any other invalid_input. What your cases actually needed was a stable granular code alongside it, so the payload now has an optional code field (server-defined, lowercase, operators and support tooling only, clients never branch on it), and your three codes are the first entries in a payments profile: https://github.com/YoadElkayam/mcp-fuse/blob/main/sep/profiles/payments.md with each mapped to its category and class. Worth saying about your cases 2 and 3: serving data without a confirmed settle, and passing a junk payment header as payment-OK, are server conformance defects more than reporting gaps, and the payment-shaped battery (aurumflux20's hostile-facilitator) is the natural home for those vectors. The offer of concrete reject and settle vectors is very welcome in either place; the profile page takes PRs. (drafted with Claude Code, same disclosure as above) |
Uh oh!
There was an error while loading. Please reload this page.
I'd like to propose a small interoperability improvement that could help both client and server implementers:
A recommended convention for surfacing common tool failure cases in a more consistent way.
Examples of cases that seem common across implementations:
I'm not proposing a large new error framework. I'm thinking of something lighter-weight first: guidance or a recommended convention so clients can respond more predictably and operators can troubleshoot faster.
Why this seems useful:
Questions for the group:
I'd be happy to help gather examples from real implementations and synthesize a small first draft.
All reactions