Seven models × two scaffolds on enterprise-mini

1 · Results

Model × scaffold scores

Seven models × two scaffolds, five independent trials per task — 50 attempts per cell, 700 graded attempts on the board. Choose a metric; click a table row for the per-task breakdown.

Cost, tokens, and calls come from the gateway log. tokens in/out and calls are per-attempt averages; solve time is the median over passing attempts (a failed run's timing isn't the latency of a solve). tokens in includes every billed input token (agents that re-read the transcript each turn show higher values). Gemma is on a flat-rate plan and shows ~ (excluded from cost ranking).

Task × model matrix

All ten tasks, hardest first (sorted by mean pass rate ascending). Each cell: trials passed out of 5 for that agent×model. avg is the row mean.

What exactly ran — models, weights, hosting

Scores depend on more than the model name: the exact build, where it's served, and any quantization or context cap. Proprietary vendors don't disclose parameter counts. The model id is the literal string the gateway routes.

Why it matters here: Gemma's 32K context cap is the direct cause of its expense-audit 0/5 (the token-heaviest task) — the model may be fine; the serving plan isn't. Nemotron runs a 4-bit NVFP4 build, so its scores measure that build, not the fp8/bf16 release — and independently re-tested below as a real agent-loop gap (§ Nemotron). Sonnet 5 prices are the introductory $2/$10 (through 2026-08); Nemotron's prices are pinned in providers.yaml because it's not in LiteLLM's price map. First-party APIs serve unquantized (or vendor-managed) builds.

Notes on the board

What the board shows

Ten enterprise tool-use tasks — each a multi-step workflow over a ~50-tool mock stack, graded on exact output artifacts — run five independent times per cell: fresh container, fresh workspace, fresh budget-capped key every time. Five clean executions per cell is a read most leaderboards skip (they run once): it separates a model that can from one that reliably does — the gap consistency measures. Two findings come out cleanly. (1) The pairing decides, not the scaffold: Nemotron 3 Ultra is 0.64 rate / 0.50 consistency on ZeptoClaw but 0.32 / 0.00 on OpenClaw; gpt-oss-120B runs the reverse — 1.00 / 1.00 on OpenClaw (~$0.04/attempt) while ZeptoClaw fails support-ticket-routing repeatedly. (2) No scaffold wins on average: paired over the 70 (model, task) cells the success-rate gap is +0.07 with a 95% CI crossing zero (p=0.18) — the interaction is the finding, not a scaffold ranking. Hover any bar for its 95% bootstrap CI — pooled claims are the robust ones; single cells read as directional. The 5-trial cut (700 attempts) is what makes all of this legible: Sonnet 5 firms up 0.80 → 0.92 while Nemotron’s lucky 0.80 settles to 0.64. Serving still caps the ceiling: Gemma’s expense-audit 0/5 on both scaffolds tracks the flat-rate host’s 32K context limit, not the weights. Success rate is the share of attempts that fully pass; consistency is the share of tasks that pass all five trials (both named for what they measure; in the literature these are pass@1 and pass^5).

5-trial matrix (per agent)

350 attempts per agent. Each cell is five dots for one task×model pairing (green = pass, red = fail). Switch agent or counting rule:

Independent evaluation

Why Nemotron 3 Ultra trails every peer

On the published board Nemotron is last on both scaffolds — ZeptoClaw 0.64 rate / 0.50 consistency, OpenClaw 0.32 / 0.00 — while Fireworks gpt-oss-120B on the same host clears ~1.0 on OpenClaw. Is that a harsh grader, or a real gap? We re-tested with a two-arm protocol that separates “can the model follow the contract?” from “does the agent loop finish the writes?”

2 · Harness

Attempt lifecycle

MAKBench runs CLI agents (ZeptoClaw, OpenClaw, Aider, Claude Code, and similar) against LLMs. Each attempt gets an isolated Docker workspace; LLM traffic goes through one gateway; grading runs in a separate trusted container. Fixtures, budgets, and images are fixed per campaign so agent×model cells are comparable. Cost is taken from the gateway call log.

New agents are a YAML file — the cli-generic adapter substitutes model and prompt and points the agent’s OpenAI-compatible env at the gateway.

id: zeptoclaw
adapter: cli-generic
image: ghcr.io/qhkm/zeptoclaw@sha256:e32bce56…b86a   # digest-pinned
command:
  - /bin/sh
  - -c
  - |
    export OPENAI_BASE_URL="$OPENAI_BASE_URL"   # points at the metered gateway
    exec zeptoclaw agent --mode autonomous -m "$(cat {prompt_file})"
env_style: openai

3 · What the agent sees

All ten tasks, exactly as they're handed over

The full enterprise-mini pack. Each tab shows the verbatim prompt the agent receives, the hidden checks that grade it (never visible to the agent), and how the published models did on it. The agent gets the prompt, a mock CLI, and seed data — nothing else.

4 · Verification & scoring

Split-phase grading

The agent runs in an untrusted sandbox; checks run in a separate MAKBench-owned container with the finished workspace mounted read-only. They do not share a process.