Why My LLM Guardrail Flagged the Right Answers (And Why I Refused to Fix It)

Benchmarking a numerical hallucination checker against a frontier API and a local 8B model taught me a harsh lesson about system design. Introduction Ask a Large Language Model (LLM) to write an executive strategy document based on your machine learning pipeline’s outputs, and sooner or later, it will invent a number. It will confidently claim a “47% lift” when your optimizer actually said 12%. In a boardroom setting, this single hallucinated metric is enough to burn the credibility of the entire dashboard. Due to this, most data teams either force a human into the loop for every generated figure, or they simply refuse to ship the LLM layer at all. I decided to take the third route while building GTM Wargame (an open source go-to-market strategy simulator). I wanted to treat “ don’t hallucinate ” as a systemic architectural property, rather than relying on prompt engineering tricks. To do this, every number the agents were allowed to use - SHAP attributions, optimizer outputs, market context was stored in a strict Ground Truth Pool. Before any text reaches the UI, a ConsistencyChecker cross-references every number the agents write against that pool. Building this guardrail was the easy part. The harder question was: How often does it actually fire in a real world scenario, and when it does, is it right? To find out, I ran 60 paired simulations pitting a frontier API model against a local 8B model, and I hand audited every flagged number against a deterministically rebuilt ground truth. The first finding was expected: the local model fabricated numbers at a materially higher rate (7.2%) . The second finding is what this article is really about: every single flag the guardrail raised against the frontier model was a false positive. The model wasn’t hallucinating. It was doing correct arithmetic that my checker simply had no way to understand. My first instinct as an engineer was to “fix” the checker. By the end of the audit, I realized those false positives are the guardrail working as designed, and expanding its matching logic to eliminate them would make the whole system less trustworthy. The guardrail pattern The guardrail’s logic is deliberately boring, which is exactly why it works: Build the Ground Truth Pool : Every number the agents might reference (SHAP values, market signals, optimizer results) gets flattened into one unified pool of floats. Let the agents write freely: The Analyst-Strategist-Manager chain produces unconstrained prose using chain-of-thought reasoning. There is no rigid JSON schema they have to fight against. Parse the output after the fact: A regex pass extracts every number the LLM generated and checks it against the pool. It allows for rounding, percentage-vs-fraction scaling (“12” for a stored 0.12), and k-notation (“176k” for a stored 176,432.11). Surface the verdict: An unmatched number is flagged as a hallucination in the UI. The user sees the warning as the system doesn’t quietly try to “fix” the figure behind the scenes. The GTM Wargame pipeline: Every layer upstream of the agents exists to populate the Ground Truth Pool that the guardrail checks against. (Image by Author) Before testing it on real models, I evaluated the checker itself as a classifier using 75 labeled cases built from authentic pipeline artifacts. The cases were split into clean cases, injected cases (seeded with unmatchable numbers), and trap cases (filled with naive-checker bait formatting like $1,234, negative signs, and k-notation of real pool values) . The checker scored a perfect 1.000 precision and 1.000 recall, with zero false positives across the 258 numbers it ran on. result = ConsistencyChecker.validate_response( text=manager_summary, shap_info=shap_values, market_context=market_context, opt_results=optimizer_output, ) # Returns: {"is_valid": False, "hallucinated_values": [47.0], "error_msg": "..."} However, a perfect score deserves skepticism. The labels for this evaluation were defined by the checker’s own matching rules, so by construction, this eval couldn’t disagree with itself. It validated everything layered around the rule - number extraction, currency and comma cleaning, sign handling, pool flattening. This however says nothing about whether the matching rule itself was the right architectural contract. That question needs real models. The Experiment: Two Models, 30 Paired Seeds (60 Runs) I ran the full pipeline (XGBoost forecasting, SHAP attributions, SciPy budget optimization, followed by the agent boardroom) across 30 identical seeds. The pipeline runs on a synthetic sales dataset generated by the repo itself, so no proprietary data is involved and every result can be rebuilt from a seed. Both models faced the exact same scenarios: Gemini (gemini-3-flash-preview, the repo’s pinned default) over the API, and Llama 3.1 8B Instruct (Q4_K_M) running fully locally through llama.cpp. These runs were recorded in July 2026 − preview models change over time, so the exact version and date matter for anyone comparing against these numbers. The checker scored the analyst and strategist nodes of every run. Before looking at the outputs, I established a strict methodological rule: A flag is not a fabrication. A flag means a number failed pool matching. That is an upper bound on hallucination, not a measurement of it. Reporting raw flag rates as “fabrication rates” is exactly the kind of unverified claim the guardrail exists to prevent. Here are the raw flag counts: Source: Image by the author. Two things immediately stood out. First, the frontier model engaged with the quantitative data far more densely, writing nearly four times as many numbers per run yet triggered fewer absolute flags. Second, 13 total flags across all runs is a small enough population to audit exhaustively by hand. Auditing the Flags: Mechanical Evidence & Judgment For every flagged run, I used an offline audit script with zero LLMs in the loop to deterministically rebuild that run’s exact Ground Truth Pool from its seed. It revalidates the stored agent transcripts, and asserts the flags reproduced exactly, ensuring I was examining exactly what the checker saw at runtime. Then, I ran two mechanical passes per flag: Notation Pass: Is the value a valid x1,000 or x1,000,000 scaling of a pool value? Derivation Pass : Does the value result from a pairwise combination of pool values (difference, sum, ratio, percentage gap, per week average, etc.)? The derivation pass comes with a built-in trap. With roughly 50 values in a pool, the number of pairwise combinations runs into the thousands. By pure chance, some derivation will land near almost any small number. Therefore, a mechanical hit isn’t enough. The final classification required semantic support from the text: the model had to explicitly name the operands it was combining. Any ambiguous flags defaulted to FABRICATED . Here are the final verdicts: Source: Image by the author. Classification of the 13 raw flags against the deterministically rebuilt ground truth pool. (Image by Author) Every single one of the local model’s 10 flags survived the audit as a genuine hallucination. Conversely, not a single one of the frontier model’s flags was an actual fabrication. The False Positives: Correct, But Derived All three of Gemini’s flags followed the exact same pattern. Let’s look at the flag from Seed 1015. The ground truth pool contained an optimized price of 535.68 and a market leader price of 715.18. The LLM strategist wrote: “ …our reliance on a 25 percent price gap relative to the leader (715.18) makes us highly vulnerable to any aggressive downward moves from the competition. ” Check the arithmetic: (715.18 − 535.68) / 715.18 = 25.1%. Both operands are real pool values. Both are named in the surrounding text. The model computed a mathematically correct, highly relevant business metric. And the checker flagged it, simply because the number “25” wasn’t explicitly stored in the original pool. Seeds 1019 and 1023 produced the same pattern with different numbers, each verifiably correct, each flagged. I call this category correct-but-derived . It’s a failure mode that standard guardrail evaluations miss entirely because the eval’s labels are defined by the checker’s own matching rules. It took real model behavior and a manual audit to expose the flaw. Why I Refuse to “Fix” It The obvious engineering patch writes itself: teach the checker arithmetic. Extend matching logic to cover pairwise derivations over ground truth pool values and all three of Gemini’s false positives would vanish. I actually built that logic (it’s the derivation pass used in the audit script), but running it offline convinced me to never put it into a live guardrail. Here is why: The collision problem: Thousands of pairwise combinations over 50 pool values create chance hits everywhere. The audit handles this by requiring semantic support from the sentence, which works when a human reads thirteen flags with full context. To prevent the checker from accidentally validating a hallucination at scale just because it mathematically collided with a random derivation, I would need to put an LLM inside the guardrail to judge the semantic context. At that point, the component whose only job is to catch LLM errors becomes dependent on an LLM being right. The asymmetry of the two failure modes: A false positive is cheap and transparent; a user sees a warning on a correct number, verifies it, and moves on. The system’s skepticism is visible. A false negative, however, is expensive and invisible: a hallucinated figure sails into an executive playbook wearing a “System Verified” badge. The entire value of a guardrail is that a green light signifies a strict, undeniable truth: this number exists in the raw data. Every time the guardrail matching contract is expanded, the value of that green light is silently diluted. So, the contract stays narrow. The checker verifies existence, not derivability. Deriving new true values is legitimate LLM behavior that falls outside the contract. It gets flagged, and the human user resolves it. The Grounding Tax Now, let’s address the local model, because this is where the guardrail proved it was heavily load-bearing. After the audit, the true fabrication rates separated cleanly: Source: Image by the author. Post-audit fabrication rates with 95% Confidence Intervals. (Image by Author) The character of the local model’s hallucinations is highly instructive. It didn’t just get numbers slightly wrong; it invented entire quantitative structures that the pipeline never produced. It hallucinated an entire ROI table − $100k spend, $500k revenue” − in a pipeline that computes neither revenue nor ROI. My personal favorite was from Seed 1017: “ Reduce budget by 20% to $X. ” The model fabricated a metric and left the template variable sitting right next to it in the exact same sentence unfilled. In my last article, I wrote about the “ Abstraction Tax ”, the throughput you sacrifice when using convenient local inference wrappers on constrained hardware. This is the other side of the coin: The Grounding Tax . A quantized 8B running on your laptop buys you complete data privacy and zero API costs. But in a highly analytical workload, it pays for that privacy with a 7.2% hallucination rate on executive-facing outputs. This in part is mitigated by using bigger models on better hardware. The guardrail firing on a live local-model run, catching an invented ROI fabrication. (Image by Author) This presents an operational paradox. The deployments that force engineers onto local models − regulated data, air-gapped systems, strict client confidentiality − are the exact environments where output verification is the hardest to outsource, and where a hallucinated number does the most damage. If your privacy requirements push you toward local inference, a grounding check isn’t just a nice-to-have feature − it is the only thing that makes the deployment defensible. Two caveats. First, this guardrail benchmarks numerical grounding against a known pool, not semantic correctness. Second, K=30 runs on a single local model quantization is enough to establish confidence intervals, but it is not a definitive industry leaderboard. The takeaway isn’t “Gemini beats Llama”; the takeaway is that fabrication rates vary wildly across deployment architectures, and you need to measure yours. Conclusion I set out to measure how often LLMs invent numbers when summarizing ML outputs, and I walked away with a system design principle I didn’t expect: the audit matters more than the guardrail . Raw metrics told a highly misleading story (0.6% vs 7.2% flag rates). Only a manual classification audit revealed the truth: a 0% vs 7.2% actual fabrication rate, where every single flag against the frontier model was a mathematically correct derivation that the checker was right to be suspicious of, but wrong to condemn. If you are building numerical guardrails for LLMs, here are the rules I’m carrying forward: A flag is an upper bound, not a claim: Audit your flags before reporting fabrication rates, and resolve ambiguities against the model. Keep the checker’s contract narrow: Verify existence, not derivability. A strict, trustworthy green light is worth the cost of a few honest warnings. Evaluate against real model behavior: The most interesting failure class in this system (the correct-but-derived false positive) was entirely invisible to my standard, perfect-scoring evaluation script. All code, raw data, and empirical setups used to generate these profiles are fully reproducible. The checker evaluation can be rebuilt offline without API keys, and every flagged transcript is available in the GTM Wargame repository . Why My LLM Guardrail Flagged the Right Answers (And Why I Refused to Fix It) was originally published in Towards AI on Medium, where people are continuing the conversation by highlighting and responding to this story.