Applied ML Research · Project Dashboard
Can a self-hosted, fine-tuned ~12B model hold its own against a frontier API on a specialized, high-stakes grounded-QA domain — at a fraction of the inference cost? The honest answer: it ties the strong small frontier, trails the full frontier on raw answering, and decisively wins calibration — knowing when to refuse.
1 · The headline result
Scored over all 1,880 test cases (1,572 answerable + 308 deliberately unanswerable). The chart on the left uses balanced accuracy — not raw accuracy, which rewards an "always answer" model on an 84%-answerable set. On the metric that can't be gamed, the fine-tune leads.
macro-average of both classes · higher is better
0 — 100%
share of 308 unanswerable rows refused · higher is better
0 — 100%
The full confusion matrix — every cell is a true partition of the 1,880 cases:
| model | true answer | false refusal | true refusal | false answer | hallucination rate | balanced acc | refuse F1 |
|---|---|---|---|---|---|---|---|
| FT-12B (ours) | 1332 | 240 | 229 | 79 | 25.7% | 79.5%best | 0.589 |
| gpt-5.4-mini | 1263 | 309 | 168 | 140 | 45.5% | 67.4% | 0.428 |
| gpt-5.4 (full) | 1556 | 16 | 48 | 260 | 84.4% | 57.3% | 0.258 |
Full gpt-5.4 wins raw accuracy (85%) — but only because it's a near-constant answerer that fabricates an answer to 84% of the unanswerable questions. In a domain where a confident hallucination is the worst outcome, the fine-tune's calibrated abstention is the property that matters.
2 · Quality, judged fairly
An LLM judge quietly favors its own model family — and three families were tangled into this experiment (the teacher that wrote the gold, the student I fine-tuned, the candidate I compared against). So the panel is deliberately bias-diversified: an OpenAI-family judge (tough on us), a teacher-family judge (kind to us), and a neutral Meta/Llama judge that touched nothing. All three vote independently on every row, in both answer orders, at full coverage (1,572 grounded rows).
| judge | win | tie | lose | bias |
|---|---|---|---|---|
| gpt-oss | 298 | 965 | 309 | +0.005 |
| opus | 267 | 1081 | 224 | +0.039 |
| llama ·neutral | 54 | 1445 | 73 | 0.000 |
| judge | win | tie | lose | bias |
|---|---|---|---|---|
| gpt-oss | 92 | 1045 | 435 | −0.148 |
| opus | 75 | 1197 | 300 | −0.073 |
| llama ·neutral | 10 | 1442 | 120 | 0.000 |
The bias readout validates the design: the OpenAI-family judge is near-neutral vs mini (+0.005) but the harshest read vs its own flagship sibling, full gpt-5.4 (−0.148). The opposite leans roughly cancel, and the family-neutral judge independently agrees with the majority both times — so the conclusion doesn't depend on which judge you trust.
3 · Independent groundedness
A standard, independent framework scoring two facets of an answer. Its judge is from the frontier's own family — so it would, if anything, favor the GPT candidates, which makes the fine-tune's faithfulness lead meaningful. Read the two together: they're driven by the same deliberate property — the model's terse, extractive style — which faithfulness rewards and relevancy penalizes.
Are the answer's claims supported by the context? Rewards an answer that stays close to the source — exactly what tight, grounded extraction does.
Does the answer address the question? It reconstructs a question from the answer, so it rewards elaboration and penalizes concision.
That concision is a feature, not the flaw the relevancy score implies. Terse, extractive answers mean fewer output tokens — lower decode latency and inference cost — with tighter grounding (the faithfulness lead). RAGAS's relevancy rewards verbosity; a production RAG copilot rewards the correct, grounded answer in the fewest tokens. The two scores are one property — decode-efficient grounding — measured from opposite ends.
The trap I almost fell into
Pulled the standard overlap metrics first, and they were thrilling: the self-hosted 12B looked like it crushed both frontiers — ROUGE-L 0.725 vs 0.556, BERTScore 0.711 vs 0.515.
It was a mirage. Those metrics reward looking like the reference answer — and my model was trained on reference answers written by a strong teacher model, so it learned to echo that teacher's phrasing. Overlap was measuring stylistic mimicry, not correctness. Every instrument above exists to replace that flattering number with one I can defend.
Before the model — the data
The training set is LLM-generated — so the real question is why you'd trust a word of it. I treated the generator as untrusted and ran every row through a chain of deterministic gates. ~8,500 examples survived; the rest were dropped, or recycled into refusal signal.
RAFT on the retriever's real output. Every example is built from the actual top-5 chunks the frozen retriever returns for that question — its real near-misses and noise — so the training distribution is the deployment distribution. The ~22% of questions whose answer isn't in the retrieved context become refusal examples: plausible, on-topic hard negatives, not synthetic swaps.
That real-miss refusal signal is where the calibration win at the top comes from.
Every row had to clear a chain of checks before it counted as grounded:
Composition: ~8,500 training rows at the natural ~78:22 grounded-to-refusal ratio (no resampling), held out by hospital-level split with zero train/test overlap — so the headline measures generalization, not memorization. The honest gap I'd close next: an explicit entailment check that each question matches its quote.
How it was built
Three decisions did most of the work — each one fixing a way the measurement could lie.
Stack: gemma-4-12b-it · full-precision bf16 LoRA (PEFT) · 8×H200 single-node DDP on Modal · LlamaIndex + Qdrant (hybrid retrieval) · bge cross-encoder rerank · MLflow tracking · evaluation via a custom bias-balanced judge panel (Anthropic Batch API), a deterministic confusion matrix, and RAGAS.