Senior AI/ML Engineer
& Software Architect

Domain-Specialized SLM Fine-Tuning — Gemma 12B (bf16 LoRA)
for Grounded RAG That Knows When to Refuse

Independent Project  ·  2025–2026  ·  bf16 LoRA on 8×H200 single-node DDP (Modal)  ·  benchmarked head-to-head vs GPT-5.4 / GPT-5.4-mini

Fine-tuned Google Gemma 12B into a domain Q&A specialist to test whether a self-hosted model can hold its own against a frontier API on specialized grounded QA at a fraction of inference cost — building the retrieval, RAFT data-generation, training, and evaluation pipeline end-to-end. The honest result: it ties GPT-5.4-mini, trails full GPT-5.4 on answering, and decisively wins calibration — knowing when to refuse.

• Calibration — the deployable edge. Over all 1,880 test cases, the 12B leads balanced answer/refuse accuracy 79.5% vs 67.4% / 57.3% and correctly refuses 74% of unanswerable questions vs the full frontier's 16% (which hallucinates an answer to 84% of them) — the no-hallucination property a high-stakes RAG copilot needs.
• Judged fairly. A bias-balanced 3-model-family LLM-judge panel (independent majority, answer-order-swapped) at full coverage (1,572 grounded rows × both frontier tiers, via the Anthropic Batch API) cancels judge self-preference; the 12B ties GPT-5.4-mini and trails full GPT-5.4 — after exposing overlap metrics (ROUGE/BERTScore) as teacher-mimicry, not correctness.
• Retriever-first, retriever-aware training. Froze a hybrid retriever (dense + BM25 + RRF + cross-encoder rerank, hit@5 ~0.78) before the generator, then built every RAFT example from the retriever's real top-5 output — its actual near-misses and genuinely unanswerable cases — so the training distribution matched deployment.
• Training architecture. Full-precision bf16 LoRA (frozen base) on 8×H200 single-node DDP (Modal); val token-accuracy 0.934; leakage-safe holdout split at the source-entity level so near-duplicates can't span splits.

MAKBench — Agent evaluation harness
Hermetic, cost-metered, reliability-first

Independent Project  ·  2026  ·  Docker sandboxes · LiteLLM gateway metering · split-phase grading · generated leaderboard  ·  any CLI agent × any LLM provider

Open-source benchmark infrastructure: evaluate any CLI agent against any LLM on versioned packs of complex, multi-step enterprise and SWE tasks — full workflows over mock tool stacks and planted-defect repos, graded on exact outcome artifacts — and get numbers you can defend. Every attempt is hermetic — fresh container, fresh workspace, per-attempt budget-capped API key, gateway-only network — so dollars and tokens are metered at the network edge, never self-reported, and grading runs in a separate trusted container the agent can never touch. Repeated independent executions per cell turn "can it?" into "does it reliably?" — the consistency and flakiness metrics leaderboards can't compute from one run. Every published number recomputes from raw attempt records.

• Controlled attempts. Fresh workspace, same fixture and git baseline, digest-pinned agent images, fixed CPU/memory and wall-clock/step budgets, gateway-only egress.
• Gateway metering. Per-attempt budget-capped virtual keys; provider secrets stay on the host; spend and tokens come from the call log (per-token vs flat-rate not mixed in rankings).
• Split-phase grading. Agent sandbox and verifier container are separate; workspace is mounted read-only for checks. Tasks fail on a pristine fixture (no-op scores zero).
• Complex tasks by design. Task packs are full workflows, not quiz items: multi-step enterprise operations over mock tool stacks (CRM, mail, calendar, ticketing, policy engines) and planted-defect SWE repos. Outcome-graded against hidden weighted checks, with an anti-triviality gate — a no-op agent scores exactly zero. A live demo board and its findings (including why the same model can be reliable on one scaffold and useless on another) are on the results page.