MAKBench — Agent evaluation harness
Hermetic, cost-metered, reliability-first
Independent Project · 2026 · Docker sandboxes · LiteLLM gateway metering · split-phase grading · generated leaderboard · any CLI agent × any LLM provider
Open-source benchmark infrastructure: evaluate any CLI agent against any LLM
on versioned packs of complex, multi-step enterprise and SWE tasks — full workflows over
mock tool stacks and planted-defect repos, graded on exact outcome artifacts — and get numbers you can
defend. Every attempt is hermetic — fresh container, fresh workspace, per-attempt
budget-capped API key, gateway-only network — so dollars and tokens are metered at the network
edge, never self-reported, and grading runs in a separate trusted container the agent can never
touch. Repeated independent executions per cell turn "can it?" into "does it reliably?" — the
consistency and flakiness metrics leaderboards can't compute from one run. Every published number
recomputes from raw attempt records.
Results & method — charts, task prompts, grading, and the Nemotron follow-up.
Open page →
- Controlled attempts. Fresh workspace, same fixture and git baseline, digest-pinned agent images, fixed CPU/memory and wall-clock/step budgets, gateway-only egress.
- Gateway metering. Per-attempt budget-capped virtual keys; provider secrets stay on the host; spend and tokens come from the call log (per-token vs flat-rate not mixed in rankings).
- Split-phase grading. Agent sandbox and verifier container are separate; workspace is mounted read-only for checks. Tasks fail on a pristine fixture (no-op scores zero).
- Complex tasks by design. Task packs are full workflows, not quiz items: multi-step enterprise operations over mock tool stacks (CRM, mail, calendar, ticketing, policy engines) and planted-defect SWE repos. Outcome-graded against hidden weighted checks, with an anti-triviality gate — a no-op agent scores exactly zero. A live demo board and its findings (including why the same model can be reliable on one scaffold and useless on another) are on the results page.
Domain-Specialized SLM Fine-Tuning — Gemma 12B (bf16 LoRA)
for Grounded RAG That Knows When to Refuse
Independent Project · 2025–2026 · bf16 LoRA on 8×H200 single-node DDP (Modal) · benchmarked head-to-head vs GPT-5.4 / GPT-5.4-mini
Fine-tuned Google Gemma 12B into a domain Q&A specialist to test whether a
self-hosted model can hold its own against a frontier API on specialized
grounded QA at a fraction of inference cost — building the retrieval, RAFT data-generation, training, and
evaluation pipeline end-to-end. The honest result: it ties GPT-5.4-mini, trails full GPT-5.4
on answering, and decisively wins calibration — knowing when to refuse.
79.5%
Balanced answer/refuse acc — best of 3
74%
Refuses unanswerable — frontier: 16%
0.925
RAGAS faithfulness — most grounded
See the full evaluation — interactive charts, the answer/refuse confusion matrix, the bias-balanced judge panel & RAGAS.
Open dashboard →
- Calibration — the deployable edge. Over all 1,880 test cases, the 12B leads balanced answer/refuse accuracy 79.5% vs 67.4% / 57.3% and correctly refuses 74% of unanswerable questions vs the full frontier's 16% (which hallucinates an answer to 84% of them) — the no-hallucination property a high-stakes RAG copilot needs.
- Judged fairly. A bias-balanced 3-model-family LLM-judge panel (independent majority, answer-order-swapped) at full coverage (1,572 grounded rows × both frontier tiers, via the Anthropic Batch API) cancels judge self-preference; the 12B ties GPT-5.4-mini and trails full GPT-5.4 — after exposing overlap metrics (ROUGE/BERTScore) as teacher-mimicry, not correctness.
- Retriever-first, retriever-aware training. Froze a hybrid retriever (dense + BM25 + RRF + cross-encoder rerank, hit@5 ~0.78) before the generator, then built every RAFT example from the retriever's real top-5 output — its actual near-misses and genuinely unanswerable cases — so the training distribution matched deployment.
- Training architecture. Full-precision bf16 LoRA (frozen base) on 8×H200 single-node DDP (Modal); val token-accuracy 0.934; leakage-safe holdout split at the source-entity level so near-duplicates can't span splits.
Laika — BYOK coding agent for VS Code
Chat-first, policy-governed autonomy
Independent Project · 2026 · TypeScript · VS Code extension host · React sidebar · in active development
A bring-your-own-key coding agent for VS Code: chat is the product, and Orbit
is a live 50,000-ft view of the in-flight change set. Built as a host-agnostic core
runtime (no editor imports), a VS Code extension host, and a React sidebar — so the
agent outlives any single editor.
- Tiered autonomy. Manual / guarded / autonomous modes governed by a per-repo policy file — guarded mode auto-allows reads while edits and unknown commands ask once; cancellation is first-click.
- Provider-agnostic BYOK. Anthropic or OpenAI keys held in VS Code SecretStorage; model registry with per-user overrides, including context-window overrides.
Conclave — Multi-expert deliberation rooms
Round-robin agents that critique until they converge
Independent Project · 2026 · FastAPI · LangGraph · React/Vite · BYOK OpenAI / Anthropic / Google
A ChatGPT-style workspace where you seat LLM experts, set a topic, and let them
deliberate in round-robin — streaming thoughts, updating a shared document,
and refining a proposal until they converge (or you pause to direct). Every expert
first writes its own version from scratch; the room then merges the way developers review
a PR — proposed changes carry reasons, each expert critiques the others' proposals,
and the merged changes execute only after true convergence. Built as a maintainable
monorepo so desktop packaging later doesn't force a rewrite.
- Democratic merging. Proposed changes carry reasons, every expert critiques them, and execution waits for the room's agreement — a peer-reviewed merge rather than a single model's synthesis.
- Orchestration. LangGraph room loop with chair turns, lap checks, binding chair direction, and a safety ceiling — convergence only when votes and proposals actually agree.
- Shared workspace. Attachments, collaborative markdown doc with live diffs in-thread, and a markdown converged-solution panel that appears only after true convergence.
- BYOK connectors. One developer API key per provider (OpenAI / Anthropic / Google) — edit, rename, delete experts and rooms without rewiring the stack.