MCP Continuous Auditor

A persistent, multi-agent auditor that continuously tests and hardens MCP servers — with promptfoo + CI as the deterministic source of truth, never an LLM's opinion.

🇩🇪 Deutsche Version

Overview

This project runs a continuous auditor for MCP servers, starting with zurich-opendata-mcp. An OpenClaw gateway exposes the auditor on Telegram as a control plane. Unlike a "vibecoding" agent, verification is a versioned artifact (pytest + promptfoo running in GitHub Actions), and a human is always the merge gate.

Features

• Read-only first — the agent reports before it ever writes.
• Deterministic ground truth — promptfoo YAML asserts + JSON-schema drift checks, run in CI.
• Recall floors, not just schema — a collapsed result set keeps its JSON shape, so a structural diff cannot see it. Probes carry min_count floors, and a second weekly probe calls the server's own tools live: the raw-URL probe verifies the endpoint, the canary verifies the whole chain.
• Identity probe — does the server report the version it actually is? A hand-maintained version in the User-Agent drifts silently: nothing breaks, no test fails. scripts/identity_probe.py reads the source and, with --installed, the shipped artifact. A portfolio sweep found 12 of 30 servers sending a wrong version, 4 of them a wrong major — docs/probes/identity.md.
• Shipped probe — is the release users install current, and does it run? CI tests the branch, not the artifact: meteoswiss-mcp shipped an import error to every fresh install for three days with main already correct. scripts/shipped_probe.py runs at two depths — --metadata-only is two HTTP requests against the Simple API (the surface pip installs from); the default depth installs into a fresh venv and speaks real MCP to it. NO_TAGS, STALE_ARTIFACT and UNCONFIRMED keep "could not measure" apart from "in sync" — docs/probes/shipped.md.
• Yank probe — the inverse question, across the whole catalogue: does a known-broken, not-yanked release still exist, with a healthy successor beside it? scripts/yank_probe.py walks every version's Requires-Dist over PEP 658 core metadata and raises UNYANKED_BROKEN_RELEASE only when four conditions hold together. It recommends a yank and never performs one — every request is a GET. All six predecessors of zurich-opendata-mcp 0.5.1 carried the uncapped range; a probe reading only latest-1 would have called the catalogue clean — docs/probes/yank.md.
• Published probe — what does the installed artifact do on the wire? swiss-efv-mcp passed identity and shipped while the package every user installs announced itself as Mozilla/5.0 … Chrome/124.0. scripts/published_probe.py installs into a throwaway venv and measures the User-Agent, the imports, the console script's start event, and the upper bounds on what it imports. Where it cannot resolve a value it reports UNVERIFIED, never clean — docs/probes/published.md.
• Lockfile probe — pyproject.toml states the bound; does the lockfile the deployment installs from state it too? The bounds PR merged green with uv.lock unregenerated: the fix was in the file everybody reads and absent from the file that installs. scripts/lockfile_probe.py compares the recorded requires-dist and the pinned versions, and asks uv lock --check / poetry check --lock where they exist. --check is hard-coded: uv lock without it rewrites the evidence.