Benchmaxx the Tools, Not the Model

omp is a terminal coding agent that made a bet most of its rivals didn’t: that the model everyone argues about is rarely what decides whether your edit actually lands. The proof is in how these agents break.

When an AI agent edits your code, what usually fails is the harness, not the model: the rigid format the model must write an edit in, the tools it reaches for, the search that either finds the function or doesn’t. The most common failure is almost comically dumb. To change one line, most agents make the model retype that line exactly so the tool can locate it. Reproduce it with a single wrong space or a smart quote and the match fails, burning a turn. On a cheap model, that happens constantly. So a reverse-engineer named Can Bölük, whose tools VTIL and NoVmp pull apart commercial software protection, took a lean, well-liked terminal agent called Pi and spent 1.14 million lines of code rebuilding its harness into a machine. The fork is now more than five times the size of the original. Part of it maintains itself.

The Premise

omp’s whole argument fits on a bumper sticker its README actually uses: benchmaxx the tools, not the model. The claim is blunt: a better edit format beats a bigger brain, so that’s where omp spends its effort.

The centerpiece is an edit format with an ugly name, hashline, and a genuinely clever idea. The model never retypes your code. Instead, every time it reads a file, omp stamps that file with a short four-character fingerprint. To make a change, the model writes something like “in the file tagged A3F2, replace line 9” and then the new line. That’s it. Two whole categories of failure vanish at once. It can’t edit the wrong copy of a duplicated line, because it named a line number. And if the file changed underneath it, the fingerprint won’t match, so omp knows the edit is stale instead of blindly applying it.

The model never retypes your code. It points at a line number and a fingerprint, and the whole “string not found” failure mode disappears.

The second thing worth knowing is who fixes omp’s bugs. A contributor called roboomp authored almost a third of the project’s commits this past June. roboomp is not a person. It is omp itself, running on its own repository: it watches the issue tracker, reproduces bugs, and opens pull requests, which a human then merges. A coding agent whose changelog is substantially written by the coding agent is either a gimmick or a sign the thing gets hammered on real work every day. The code says the latter.

The Machine

Under the hood, omp is one agent loop behind several front-ends: a terminal UI, a one-shot command, a scriptable server, the self-fixing bot, all feeding the same engine. Three choices give it its character.

It went native. The hot path (search, file-walking, the shell, syntax parsing) is roughly 53,000 lines of Rust that the JavaScript calls through a real native bridge, not a slow shim. Its grep is literally ripgrep’s own engine running in the same process, so there’s no cost to spawn a subprocess every time the agent wants to look something up. It even embeds an entire bash implementation so shell commands don’t fork out to your system.

It went wide. Fifty-eight model providers and over 3,600 models are catalogued in the box, from the big labs down to whatever you’re running on localhost. Pointing it at a custom endpoint is a few lines of config. And it reaches into the parts an IDE usually owns: real language-server diagnostics, syntax-tree-aware edits, a debugger interface, and code-execution kernels for Python and JavaScript that call back into the agent’s own tools mid-run. “Batteries included” undersells it. There are more of them here than anything this size ships with.

The Test Drive

We didn’t want to grade omp on its own benchmarks, so we built a fake model. A small mock server plays the part of the LLM, returning exact scripted tool calls, and we check what actually hits the disk. Everything below is deterministic, costs nothing in API fees, and is reproducible by anyone.

The edit format is the real thing. We threw fifteen nasty scenarios at it: five identical lines with an instruction to change only the third, emoji and Chinese characters around the edit, Windows line endings, a file mutated behind the agent’s back. All fifteen landed byte-exact. Better, when we told it to edit a line the model had never actually been shown, it refused, with a little lecture and the real contents of those lines. Run the same trick against upstream Pi and the difference is sharp but fair: Pi lands the same correct edit, but it has no idea the file drifted underneath it. omp warns you; Pi is simply blind to the question.

We built a fake model to grade the real one. Fifteen adversarial edit scenarios, all byte-exact, and it refused to edit code it had never been shown.

Speed is more nuanced than the marketing. omp’s search is genuinely fast, about 2× quicker than spawning ripgrep on a normal-sized repo, because it never pays the startup cost. But on a giant 81,000-file monorepo, plain ripgrep pulls ahead by up to 1.5×. And we found a real bug: omp’s native search silently stops reading a file after 4 MB, reports the file as fully searched, and returns nothing. On a large log or generated file, it will miss matches and never tell you. The measured version is narrower than “fastest in the west”: fastest for the everyday case, with a sharp edge on big files.

Then there’s the weight. The install is 995 MB, and about 39% of that is optional machine-learning and text-to-speech models that are turned off by default. You’re carrying a third of a gigabyte you almost certainly won’t use, and, annoyingly, there’s no supported way to trim it. Every launch also pays about a second of startup just parsing its own bundle, and a trivial task runs roughly twice as slow as it does on the leaner Pi. Fast where it counts, heavy everywhere else.

The Fine Print

Out of the box, omp runs whatever the model says, no questions asked. Its default approval mode is called yolo, and the name is accurate. We confirmed it: a fresh install will execute rm -rf, run arbitrary shell, and make network calls with zero prompt. Its own detector for catastrophic commands is switched off by that default. And file writes aren’t fenced to your project, so a misdirected edit can land a folder above where you’re working.

This matters because omp, like every agent of its kind, reads files from the project it opens as instructions, including an AGENTS.md it finds lying around. Clone a hostile repo, open omp, and a booby-trapped instruction file can tell the agent to do something ugly, which the default settings will then do without asking. None of this is malice or a hidden backdoor; we looked, and the supply chain is clean, the collaboration crypto is sound, and there’s no sneaky phone-home. The problem is that the safe posture is one config line away and isn’t the default.

Its default mode is called yolo, and the name is accurate: it runs rm -rf with no prompt. The safe setting is one line away. It just isn’t the one you start with.

One more piece of honesty. omp’s most quotable claims are the reason people try it: a cheap model’s success rate leaping from 6.7% to 68.3%, another cutting its output by 61%. We could not reproduce them. They need paid model APIs, and they were measured on an earlier version of the edit format than the one that ships today. We verified that the mechanism those numbers credit is real and works. We did not verify the numbers. Anyone who tells you otherwise is selling something.

The Playbook

Start with the edit format, because it costs nothing and it’s the part we’re surest of. Point omp at the cheapest model you can stand, leave the default hashline mode on, and hand it a real editing job: a rename that touches twenty files, or a change buried in a function full of near-identical lines. You’re testing the format here, not the model: whether it keeps a weak model out of the ditch. On our bench it did, across fifteen scenarios built to break it. An afternoon, and you’ll know whether a cheap model plus a good harness covers work you’ve been paying a frontier model to do.

Give it a weekend and wire in the parts an IDE usually owns. Install a language server so the agent reads real type errors instead of guessing, then try ast_grep for structural search and the two-step ast_edit for structural rewrites. While you’re in there, steal the trick this whole review is built on: stand up a fake model, a few dozen lines of a mock server that replays scripted tool calls, and use it to test any agent’s edit-and-run loop for free and deterministically. It is the cheapest way to learn whether a harness does what it claims before you wire it into CI, and it caught real bugs in omp that a live model would have hidden in noise.

Run one setup step before any of it, though: take omp off yolo. The default runs whatever the model says, rm -rf or a stray curl included, with no prompt. So set tools.approvalMode: write, treat a cloned repo’s AGENTS.md as untrusted input, and keep genuinely unknown code in a container. Do that once and the longer bet opens up. Point the roboomp pattern at your own issue tracker and let an agent reproduce, fix, and open the pull request while a human keeps the merge button. omp already runs that play on itself every day, in the open, and lets you read exactly how. So pick one play for the weekend, make it the edit-format test, and let the rest wait until you’ve watched a cheap model land its edits clean.