Most teams building LLM products treat evaluation as an afterthought: a vibe check, a handful of manual spot-reviews, maybe an LLM-as-judge bolted on late because someone read a blog post about it. I learned to take it more seriously the hard way, across two different production systems, a legal AI pipeline where I was the sole engineer, and a coaching platform I'm building solo now. Same underlying discipline, two different domains, and one failure mode (a bad judge quietly producing false confidence) that I don't think gets talked about enough.
The legal work came first, and it's where the discipline started: grounding evaluation in real domain-expert judgment instead of guessing. The coaching platform is where I found out what happens when you don't check whether your automated judge is actually any good.
Start with ground truth that's actually grounded
Before I trusted any automated evaluation, I needed to know what "good" meant, concretely, per document type. So I worked with practicing attorneys, through a connection to our CEO's partner's law firm, to produce gold-standard reference summaries for each of the 16 types the pipeline handled. Not a rubric written by an engineer guessing at what lawyers care about. Actual attorneys, doing the work, producing the artifact a domain expert would consider correct.
I then manually curated each component of a generated summary against those gold-standard documents, checking not just "is this roughly right" but which specific elements a rubric needed to weight, and where models tended to drift: dropped facts, hallucinated dates, misattributed testimony, structural omissions specific to each document type. That process is what turned "we have an LLM summarizer" into "we have per-document-type rubrics grounded in what a domain expert actually produces."
This is the part that's easy to skip and expensive to skip. Without attorney-produced ground truth, any evaluation system, automated or manual, is just checking a model's output against another model's assumptions about what lawyers want. That's not evaluation. That's guessing with extra steps.
Routing wasn't one-size-fits-all either
The same discipline showed up in how the pipeline chose models, not just how it scored them. CaseMark's summarization ran across Claude 3.5 Sonnet, GPT-4o, and Gemini 2.5, routed per document type and document size rather than defaulting to one model everywhere. Larger documents, the hundreds-of-pages medical chronologies, routed to Gemini specifically to take advantage of its larger context window; everything else relied on manually curated chunking strategies tuned per document type, so a deposition and a trial summary weren't being split the same naive way. Model routing and ground-truth grounding were the same underlying habit: don't default to convenient, check what the specific case actually needs.
Where I found the failure mode that mattered most
The sharpest version of the judge-validation lesson showed up later, building Threshold, the coaching platform I'm developing now. The system grades LLM-generated coaching content, morning/evening briefings, plan adaptations, workout and strength enrichments, against frozen fixtures every time generation code or prompts change, because the normal test suite mocks the LLM entirely and gives zero signal on content quality.
It splits into two tiers, gated differently because they fail differently. Objective properties (did the model cite a pace or biometric number that isn't actually in the context? did it push training load upward on a bad-readiness day when policy says it shouldn't?) are checked in code, and any violation hard-fails that fixture immediately; that's affordable because a deterministic check is reproducible. Subjective properties (does this lead with the right signal, does it read like a coach) go through an LLM judge scoring 1 to 5 against a hand-written rubric, and no single judge score is allowed to fail anything on its own. Scores accumulate into a per-surface aggregate checked against a committed baseline with a tolerance band, so one wobbly score can't false-block a deploy but a genuine broad regression still trips the gate.
The judge failing wasn't hypothetical
The original judge model was gpt-4o-mini, picked for cost. It didn't fail on style, it failed on reading comprehension, repeatably: it claimed no sleep data existed when a score was sitting right in the context, inverted pace comparisons and called 7:55 faster than 7:10, recomputed a weekly mileage total a deterministic check already owned, summed exercises across what were supposed to be standalone alternative workouts, hallucinated a dietary constraint that was never set, and scored two identical outputs a 5 and then a 2 back to back. Every one of those is a domain-specific reasoning failure, you only find them by checking the judge against cases where you already know the right answer, not by trusting that it produces a number.
It was retired from grading entirely. Not downgraded, removed.
The insight that surprised me: the judge doesn't separate good models from bad ones
Later, comparing candidate models for the generation side, the plan was to run six candidates across dozens of fixtures with the judge held fixed as a control and see which one scored highest. That's not what actually decided it, and I'd anticipated part of why: a judge fixed to one model family is a biased ruler, not a neutral one, it tended to inflate other candidates slightly and was harder on its own family's output. So rankings were built to lean on the hard gate, not raw judge means, and that turned out to be exactly where the real signal was. Judge scores came back at rough parity for every candidate regardless. What separated the field was the deterministic tier: one candidate fabricated a pace twice, another emitted malformed JSON and broke a hard safety invariant on a severe-readiness fixture, another stayed parse-fragile on one surface and fabricated a heart rate. The eventual winner was the only candidate with zero hard-gate failures, at roughly a fifth the cost of the incumbent.
That's a real lesson, not just a cost win: when you're choosing between models for a task with real correctness requirements, a judge, especially a biased or same-family one, is the wrong instrument for telling candidates apart. The reproducible, code-level checks are what actually discriminate. The judge's job is to tie-break quality once the hard gate has already done the real work.
Replacing the judge itself got the same scrutiny as replacing a generator
When it came time to move the judge seat to a newer model, it didn't get a vibe check either. The method: hold generation completely fixed, vary only the judge, and diff every verdict between the incumbent and the candidate, pass/fail agreement, score deltas, and critically, a hand-read of every single disagreement, because the real question was never "do the numbers match," it was "does the candidate misread anything the way the retired model did." The candidate agreed with the incumbent on 87% of verdicts with zero misreads in the disagreement dump (the gaps were defensible differences in judgment, not comprehension failures), and only then did it take the seat, at lower cost than the model it replaced.
A bad judge is worse than no judge
That's the throughline, and it's not an abstraction here, it's a documented failure catalog. An evaluation system that scores confidently and wrongly is worse than no automated evaluation at all. No judge means you know you're flying blind. A bad judge means you think you have coverage while it repeatably misreads exactly the cases that matter, and every one of those misreads would have shipped silently if nothing had checked the judge against ground truth first.
Even a good gate goes stale if you don't maintain it
One more failure mode worth naming, because it's easy to assume "validate the judge once" is the whole job: rubrics encode product policy, and policy moves. When a new fitness model shipped that made a certain message type a simple status update rather than a plan change, the judge kept grading it against the old rule and scored a genuinely correct output 2 out of 5, not because the judge misread the output, but because the rubric hadn't been told the policy had changed. The fix wasn't to soften the gate. It was to teach the rubric the new policy, the same way you'd patch a bug, and the surface went from failing to a clean pass once it actually reflected what "correct" meant now.
The rule that keeps this honest, and the one I'd defend hardest: when a run goes red, the only two acceptable outcomes are "fix a real bug in the generator" or "fix a real bug in the eval." Widening a whitelist, lowering a baseline, or softening a rubric to make a failure disappear isn't an eval fix, it's fraud against your own gate, and it defeats the entire point of having one.
The pattern generalizes
Two different projects, two different mechanisms: legal documents graded through my own manual curation against attorney-produced gold standards, endurance coaching graded through a two-tier gate where the judge model itself was validated, retired, and replaced on evidence. But the same underlying rule held in both, and it matters more, not less, as more of the pipeline gets automated: don't trust an evaluation system, human-built or model-built, until you've checked it against ground truth someone with real domain expertise would recognize as correct. And once you've built that trust, don't assume it's permanent. A gate that isn't maintained against a moving target is just a rubric quietly grading against yesterday's rules.
If you're building or reviewing an LLM-as-judge system, the one question worth asking before you trust any of its output: has anyone actually validated the judge against ground truth a domain expert produced, or are you trusting it because it produces a number?
Those are different systems, even when they look identical from the outside.