o2-mcp

Generic, project-agnostic access to the HMS O2 cluster, exposed both as a Python library (o2mcp) and as an MCP server (o2-mcp) so an agent can submit Slurm work, run remote commands, monitor jobs, move files, and keep disk tidy — without triggering a Duo push on every action.

Extracted from clock-oscillation-analysis so the cluster tooling is shared infrastructure (used by multiple analysis projects) rather than living inside one of them. Project-specific layers (e.g. run-organization for a particular pipeline) build on this package rather than living in it.

Duo model (read this first)

Our August 2026 incident testing showed that HMS O2 can issue Duo challenges not only for a new SSH transport, but also when a new session channel is opened inside an existing OpenSSH ControlMaster. A normal ssh o2 command invocation therefore is not Duo-safe merely because it reaches the expected mux socket.

Version 0.4 replaces that command pattern with one workstation-wide broker per configured host role. Login-node work normally needs only the login broker; a separate transfer broker exists for commands that must execute on the transfer host:

1. o2_start_broker consumes a short-lived, client-bound, one-attempt role-matched grant and starts exactly one SSH process for that broker.
2. That SSH process opens exactly one remote session running a small embedded Python helper.
3. Every later o2_exec, Slurm, workspace, keepalive, and run-organization command is length-prefixed JSON sent through the same session's stdin/stdout.
4. Independently launched MCP tasks share the applicable broker through a mode-0600 Unix socket. Commands are serialized within each role in the MVP.
5. The daemon never reconnects. If the channel dies, its socket disappears and later commands fail locally; another login requires another explicit grant.

The workstation-wide ~/.agent_locks/O2_POLICY.json still defaults fail-closed. Both the MCP client and the broker daemon check it before every logical command, so a concurrent disabled transition wins even against a hand-crafted local socket request. Disabling does not terminate a command already in progress.

The broker uses an O2B1 magic marker, four-byte network-order lengths, and UTF-8 JSON rather than newline framing. Remote stdout/stderr is drained while retaining at most 1 MiB per stream, so noisy commands cannot grow broker memory without bound and newlines or JSON-looking output cannot corrupt the next command. Frames are limited to 16 MiB, while command text is capped at 64 KiB so bash -c stays below the remote execve argument limit. One command timeout returns code 124 without reconnecting or replacing the persistent channel. Finite deadlines are capped at seven days to avoid platform socket overflow; an explicit None retains indefinite execution. Logical commands use a non-login Bash that inherits the one session environment, avoiding repeated profile banners and startup latency. Full escaped requests are size-checked before dispatch, stdin is capped at 1 MiB, and transport writes use inactivity plus frame-size-scaled deadlines so a slow progressing link survives while a stall cannot retain the global policy mutex indefinitely. For finite commands, the daemon also bounds the result-frame wait with an inactivity timeout and a frame-size-scaled absolute budget; slow progress survives, while a silent helper terminates and unpublishes the one transport instead of leaving a falsely reusable broker wedged.

ControlMaster hardening remains in the library for the transfer compatibility layer and offline regression tests. It is not the login command boundary: both the MCP wrapper and public O2Connection.start_master() API reject login master starts and raw SSH, directing callers to the broker instead. See the broker design.