Benchmaxx the Tools,
Not the Model

Weaker, cheaper models burn turns on the usual edit format: they retype a line of your code to find it, get one space wrong, and the match fails. Retry, repeat, pay again.

omp's hashline format never asks the model to retype old code. It anchors each edit to a line number and a short hash of the file, so "string not found" stops being a failure mode.

[auth.ts#A3F2]        # tag minted when the model read the file
SWAP 9.=9:            # replace line 9 — no old code retyped
  return verify(pw)

Across 15 adversarial edit scenarios — five identical lines, emoji and CJK, Windows line endings, a file changed mid-edit, a 3,000-line file — the format landed all 15 byte-exact, and refused edits to code the model had never actually been shown.

The eye-popping score jumps omp advertises (one cheap model going 6.7% → 68.3%) are its own numbers, measured on an older version of the format. We verified the mechanism, not the scores — that needs paid model APIs.

You juggle Claude, GPT, a local Llama, and a work endpoint — each behind its own tool, each configured differently.

58 providers and 3,600+ models ship in the box, and a custom OpenAI-compatible endpoint is about six lines of YAML — with the API key optional.

# ~/.omp/agent/models.yml — a keyless local endpoint
- id: my-endpoint
  baseUrl: http://localhost:8080/v1
  api: openai-completions
  auth: none          # API key optional

We drove the shipped binary against a homemade endpoint with a fake key and no internet — one models.yml block, api: openai-completions, auth: none — and ran full agentic turns through it. It just worked.

58 providers / 3,607 models catalogued in-box against the README's advertised "40+" — the breadth is real, and then some.

Your agent guesses at types and greps blind, because all it really has is text search and hope.

omp gives the model real language-server diagnostics, structural (syntax-tree) find-and-replace, and search that runs ripgrep in-process instead of shelling out.

$ omp lsp diagnostics src/broken.ts
7:8 [error] [typescript] Type 'string' is not
   assignable to type 'number'. (2322)

Headless, it returned compiler-grade errors — "Type 'string' is not assignable to type 'number'", code 2322, exact line and column — did structural edits, and its search beat spawning ripgrep by 2.1× on a typical 5k-file repo.

Bring your own language server — none is bundled. And native search silently stops reading a file past 4 MB, a real blind spot on large generated or data files.

You want to know whether an agent's edit-and-run loop actually holds up, but real models are non-deterministic and every test run costs money.

Script a fake model that returns exact tool calls, point the agent at it, and assert on what actually lands on disk. Deterministic, free, repeatable.

omp --model mock/mock-model -p "fix the bug" --mode json
# assert on disk state, not stdout vibes
# SSE cut mid-tool-call → exits 0 with empty stdout (bug)

This entire review ran that way: a zero-dependency mock server driving omp through scripted read → edit → run turns, checking byte-exact file state. It caught real bugs — including a mid-stream network drop that makes the agent exit successfully with no output.

It tests the machinery — does the edit land, does the retry back off — not how smart a real model is on your actual code.

Issues pile up faster than anyone triages them, and most are small, reproducible, and boring to fix.

The roboomp pattern: an agent watches the issue tracker, reproduces the bug in an isolated worktree, fixes it on a branch, opens a PR — and a human merges.

issues.opened → classify → reproduce in worktree
  → fix on branch → open PR   # gh-proxy alone holds the PAT
human reviews & merges     # roboomp never self-merges

omp already does this to itself. Its roboomp account authored 15% of all commits in the repo's history and 29.5% of June 2026's — agent-written, human-gated. It opens the PRs; it never merges its own.

With the default settings, an untrusted issue is a command-execution risk — the bot isolates credentials behind a proxy precisely because the agent underneath does not sandbox itself.