Skip to content

Bump conformance harness to 0.2.0-alpha.11 #3282

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
maxisbey merged 1 commit into from
Aug 11, 2026
+19 −2
Merged

Bump conformance harness to 0.2.0-alpha.11 #3282

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 18 additions & 1 deletion .github/actions/conformance/client.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,16 +11,18 @@
--spec-version is omitted the harness picks per-scenario (LATEST_SPEC_VERSION
for active scenarios, DRAFT_PROTOCOL_VERSION for draft-only ones).
- Server URL as last CLI argument (sys.argv[1])
- Must exit 0 within 30 seconds
- Must exit 0 within the harness --timeout (CI passes 60s; the default is 30s)

Scenarios:
initialize - Connect, initialize, list tools, close
tools_call - Connect, call add_numbers(a=5, b=3), close
sse-retry - Connect, call test_reconnection, close
json-schema-ref-no-deref - Connect, list tools (no $ref deref)
json-schema-2020-12-preservation - List tools, echo the focal inputSchema back verbatim
request-metadata - Connect with all callbacks; client stamps _meta
http-standard-headers - Connect, call a tool (Mcp-* headers checked)
http-invalid-tool-headers - List tools, call every surfaced tool (x-mcp-header filter)
http-custom-headers - Replay the harness's toolCalls (x-mcp-header -> Mcp-Param-*)
elicitation-sep1034-client-defaults - Elicitation with default accept callback
sep-2322-client-request-state - Drive the MRTR auto-loop (SEP-2322)
auth/client-credentials-jwt - Client credentials with private_key_jwt
Expand Down Expand Up @@ -252,6 +254,21 @@
await client.list_tools()


@register("json-schema-2020-12-preservation")
async def run_json_schema_2020_12_preservation(server_url: str) -> None:
"""List tools, then echo the focal tool's inputSchema back verbatim (SEP-1613 / SEP-2106).

The harness diffs what the client round-trips through `json_schema_echo` against its
fixture to detect 2020-12 keywords ($schema, $defs, $anchor, additionalProperties,
allOf/anyOf, if/then/else) being stripped while parsing tools/list. Unlike
json-schema-ref-no-deref, this mock is version-aware, so client_mode() applies.
"""
async with Client(server_url, mode=client_mode()) as client:
listed = await client.list_tools()
focal = next(tool for tool in listed.tools if tool.name == "json_schema_2020_12_tool")
await client.call_tool("json_schema_echo", {"schema": focal.input_schema})

Check warning on line 269 in .github/actions/conformance/client.py

View check run for this annotation

Claude / Claude Code Review

[quality] Missing-focal-tool failure surfaces as opaque coroutine-StopIteration 

The bare `next()` on line 268 has no default, so if `json_schema_2020_12_tool` is ever absent from the `tools/list` result (a future harness pin bump renaming the fixture, or the SDK's x-mcp-header filtering dropping it), the escaping `StopIteration` surfaces as an opaque `RuntimeError: coroutine raised StopIteration` that names neither the missing tool nor the listing. Consider `next((t for t in listed.tools if t.name == "json_schema_2020_12_tool"), None)` followed by an explicit `RuntimeError`
Comment on lines +267 to +269

@claude claude Bot Aug 11, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 The bare next() on line 268 has no default, so if json_schema_2020_12_tool is ever absent from the tools/list result (a future harness pin bump renaming the fixture, or the SDK's x-mcp-header filtering dropping it), the escaping StopIteration surfaces as an opaque RuntimeError: coroutine raised StopIteration that names neither the missing tool nor the listing. Consider next((t for t in listed.tools if t.name == "json_schema_2020_12_tool"), None) followed by an explicit RuntimeError listing the surfaced tool names, matching this file's existing precondition-guard convention.

Extended reasoning...

What the issue is. run_json_schema_2020_12_preservation locates the focal tool with a bare next(tool for tool in listed.tools if tool.name == "json_schema_2020_12_tool") — no default argument. When the generator is exhausted without a match, next() raises StopIteration. Because this happens inside a coroutine, PEP 479 semantics kick in: a StopIteration escaping a coroutine frame is converted by the interpreter into RuntimeError: coroutine raised StopIteration. The resulting traceback names neither the missing fixture tool nor the tools/list result — it reads as interpreter internals, not as "the harness fixture wasn't found".

The code path that triggers it. The scenario handler lists tools via client.list_tools() and then searches for the harness-owned fixture tool by name. Two realistic triggers can make that search come up empty: (1) a future harness pin bump renames json_schema_2020_12_tool — this file exists precisely to track the pinned harness, and the workflow comment (conformance.yml) instructs bumping deliberately and reconciling baselines, so pin bumps are this file's expected lifecycle; (2) the SDK's own tool filtering drops the tool from the listing — run_http_invalid_tool_headers in this same file documents that the SDK filters tools with malformed x-mcp-header annotations out of list_tools results, so a filtering change in _absorb_tool_listing (src/mcp/client/session.py) is a second real path to an empty match.

Why existing code doesn't prevent it. Nothing guards the lookup. The rest of the file follows an explicit-diagnostics convention for missing preconditions — there are multiple raise RuntimeError("MCP_CONFORMANCE_CONTEXT missing 'client_id'")-style guards — but this handler skips that pattern for the one precondition it has.

Impact. CI-only. On the happy path (harness alpha.11 as pinned, where the PR reports 9/9 passing) nothing goes wrong. In the failure scenario the leg goes red either way; only the diagnostic quality differs — whoever does the next pin bump has to reverse-engineer coroutine raised StopIteration from a solo-rerun log instead of reading a message that points at the fixture and shows which tools were surfaced (which would also distinguish 'harness renamed the fixture' from 'SDK filtered the tool').

Fix. Identical passing-path behavior, strictly better failure diagnostics:

focal = next((t for t in listed.tools if t.name == "json_schema_2020_12_tool"), None)
if focal is None:
    raise RuntimeError(
        f"json_schema_2020_12_tool not in tools/list result: {[t.name for t in listed.tools]}"
    )

Step-by-step proof. Suppose harness alpha.12 renames the fixture tool to json_schema_2020_12_focal. (1) CI bumps CONFORMANCE_PKG; the scenario runs and listed.tools contains json_schema_2020_12_focal and json_schema_echo but no json_schema_2020_12_tool. (2) The generator inside next() is exhausted with no match, so next() raises StopIteration. (3) The exception propagates out of the coroutine frame run_json_schema_2020_12_preservation; per PEP 479, the interpreter replaces it with RuntimeError: coroutine raised StopIteration. (4) asyncio.run(handler(server_url)) in main() propagates that, the client process exits non-zero, and the harness records a failure whose only Python-side evidence is the interpreter-internals message — no mention of the tool name or the listing. With the fix, step (3) instead produces RuntimeError: json_schema_2020_12_tool not in tools/list result: ['json_schema_2020_12_focal', 'json_schema_echo'], immediately identifying the rename.

@maxisbey maxisbey Aug 11, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

doesn't seem like an issue for a conformance test?



@register("tools_call")
async def run_tools_call(server_url: str) -> None:
"""Connect, list tools, call add_numbers(a=5, b=3), close."""
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/conformance.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ env:
# Pinned conformance harness package spec (passed verbatim to `npx --yes`).
# Bump deliberately and reconcile both
# .github/actions/conformance/expected-failures*.yml files in the same change.
CONFORMANCE_PKG: "@modelcontextprotocol/conformance@0.2.0-alpha.10"
CONFORMANCE_PKG: "@modelcontextprotocol/conformance@0.2.0-alpha.11"

jobs:
server-conformance:
Expand Down
Loading