The Harness Above the Harnesses

Databricks started this two weeks before we first put it on the bench. In that stretch they wrote 568,000 lines of code, put the creators of Spark and MLflow on the launch, and made a bet most coding-agent startups would find rude: the layer worth owning isn’t another agent, it’s the thing that governs all of them. One YAML spec drives Claude Code, Codex, Cursor, or Pi. A policy engine rules on every action they take. Start a session on your laptop and it follows you to the browser, then to the phone.

The Premise

The word Databricks wants is meta-harness, and the promise under it is narrow enough to check. Your agent spec holds no harness-specific code, so changing one field changes the backend. Above the spec, a governance layer rules on every action: allow, deny, or ask. The load-bearing part is that an agent cannot weaken its own governance from the inside. That is the claim we kept testing.

The fleet grew from 11 harnesses to 17 across two releases — the holdout is Databricks’ own.

The Machine

Four layers, stacked. At the top, clients: a CLI, a mobile-first web UI, a native macOS app, all speaking to one session. Beneath them a FastAPI server on Postgres exposes 58 API paths behind real auth (argon2id, JWT, OIDC, three-level RBAC) and streams events over SSE and WebSockets. Then the policy engine: two dozen handlers across six phases, evaluated stricter-wins, so a DENY short-circuits anything softer. At the bottom, the meta-harness runs each executor one of two ways — in-process through a vendor SDK, or by driving the real vendor CLI through terminal emulation inside an OS sandbox. A secretless credential proxy sits in that second path, so a child agent only ever sees a placeholder where a real key would be.

The Test Drive

We benched it twice. June came first, against 0.2.0: five deterministic, model-free trials. Install took about six seconds and cost a 458 MB virtualenv. The policy engine decides, before an agent acts, whether to allow the action, ask you, or deny it, so we fed it fifteen of the calls a real fleet hits: an over-budget model, a model it had never seen, a raw shell command, an agent trying to grant itself new powers. It ruled the cautious way on every one — the expensive model denied, the unknown one refused rather than waved through, the self-promotion blocked. Run the whole engine at once and a hard DENY still overrode a softer ASK, even one declared first. The single blemish was the backend list, where the validator accepted only 11 of the 18 agents we tried while the README promised more. We filed it as version skew; a fix for the exact skew we’d surfaced landed upstream within the hour.

July was the re-run, against 0.4.0, three passes with byte-identical output. Our June scripts needed no changes at all two releases later, which says something quiet about the API under the churn. Empty directory to version banner: 7.15 seconds, three steps. Every one of those policy calls came out the same way it had in June. The surprise was the backend list: the same agent spec now validated against 17 of the 18 agents we tried, up from 11. Every one June had filed under “wait,” OpenCode and Goose and Qwen and Copilot and Hermes, now worked, and the allowlist had quietly grown two families nobody announced, kimi and kiro. The re-bench cost zero tokens. The whole suite is model-free, and it ships in the study as recipes you can run yourself.

The Fine Print

The scan came back clean: no critical issues, no backdoors. A few things are worth admiring. The sandbox fails loud, raising an error rather than quietly dropping to no isolation. The credential proxy keeps real keys away from child agents. There is no third-party analytics anywhere in the tree. Two cautions, though. A starlette pin holds back the fix for six known vulnerabilities on the server path. And the OS sandbox is opt-in per spec — the flagship orchestrator ships with sandbox: none and leans on git worktrees and a blast-radius policy instead.

Agents cannot silently weaken their own governance.

The Playbook

Start with the referee. It is an afternoon of work. Declare a cost_budget and one permission policy, and from then on every model call and every tool invocation, from any harness, is ruled on before it runs. This is the best-evidenced thing in the system: across two releases and three runs each, every governed decision we tried came out right — a DENY overriding an ASK that was declared first, an unrecognized model turned away at the door, an agent’s bid to widen its own limits refused. Version 0.4.0 adds per-subagent budgets, which is the control you want before a coordinator starts handing your key to workers it spawned.

Give the fleet itself a weekend. Write one agent body, swap executor.config.harness, and run it on Claude Code, then Codex, then Cursor. Seventeen of the eighteen agent backends we tried accepted that one spec on our bench, so portability here has stopped being a roadmap line. It shipped. Once two vendors are live, steal Polly’s trick even if you never touch the server: fan the work across git worktrees, then hand one vendor’s diff to another for review. The shipped example loads and validates under the pinned release. What no one has measured, us included, is whether that cross-vendor review is any good, so the play sits at demonstrated rather than proven.

The pocket session and the synthetic keys are promissory notes. Both are well-built where we could read the source, and neither was exercised on the bench; the tiers above say as much. So run one play tonight, and make it the referee. It is what the rest of the fleet stands on, and the only blank check it leaves open is the one you sign yourself.