100% Recall, 38.5% Precision: What Happened When My AI Auditor Audited Itself

Building a review-gated, tamper-evident audit pipeline for agentic AI and discovering that the auditor was not ready for production The most useful output from my AI-auditing system was a red status label: Requires Review. ThirdLine had caught all five defects I deliberately planted in a synthetic fleet of banking agents. Recall was 100%. On the deterministic evaluator, F1 reached 0.909. Then I ran the audit end to end with GPT-4o-mini on the same test fleet. Precision fell to 38.5%, F1 fell to 0.556, and the system’s own model card failed it against the threshold I had written. ThirdLine is a system that audits other AI agents, checking them for hallucination, bias, drift, and security failures before those failures reach production. This piece walks through how the audit pipeline and its human-review gate work, and what the results revealed about the limits of using AI to audit AI. That failure became the real project. ThirdLine is open source under the MIT license, the full implementation discussed below is available on GitHub. The governance gap is narrower and stranger than it sounds On April 17, 2026, the Federal Reserve, OCC, and FDIC issued revised model-risk guidance. A footnote explicitly states that generative and agentic AI are outside its scope because they are novel and rapidly evolving. The same footnote immediately adds that banks should still use their risk-management and governance practices to determine appropriate controls. That is not an exemption from governance. It is a gap in prescriptive model-validation guidance. Traditional validation works best when the object under review is comparatively stable: defined inputs, defined outputs, a measurable error distribution, and a controlled release process. An agent can retrieve different context, change the prompt it sends downstream, call tools, retry failed steps, and follow a different execution path for the same user request. The unit under audit is no longer just a model. It is a system with state, permissions, dependencies, and runtime behavior. I built ThirdLine to test one possible response: an orchestrator and five specialist evaluators auditing five synthetic banking agents across hallucination, bias, drift, robustness, and reliability. The pipeline discovers the fleet, collects evidence, evaluates interactions, maps failures to controls, drafts findings, and places every finding into a human review queue. The obvious objection is that an AI auditor can be wrong too. I agree. The design only makes sense if the auditor can produce evidence but cannot grant itself authority. Figure 1: ThirdLine’s six-step audit pipeline, from fleet discovery to the human review queue. Image by author. A human-in-the-loop gate has to be more than a warning Many agent workflows call something “human in the loop” when they really mean “log a warning and continue.” That is monitoring, not a control boundary. In ThirdLine, the orchestrator’s final step writes each finding to a queue with a PENDING status and a severity-based service-level deadline. The orchestration component contains no branch that changes the finding to APPROVED. def _step_hitl_gate( self, findings: list[dict], run_id: str, ) -> list[dict]: queue_entries = [] for finding in findings: sla_hours = { "CRITICAL": 4, "HIGH": 24, "MEDIUM": 72, "LOW": 168, } now = datetime.now(timezone.utc) deadline = now + timedelta( hours=sla_hours.get(finding["severity"], 24) ) entry = { "queue_id": str(uuid.uuid4()), "finding_id": finding["finding_id"], "severity": finding["severity"], "status": "PENDING", "sla_deadline": deadline.isoformat(), "run_id": run_id, } queue_path = ( self._queue_dir / f"{entry['queue_id']}.json" ) queue_path.write_text( json.dumps(entry, indent=2, default=str) ) queue_entries.append(entry) return queue_entries The important choice is not the JSON file. It is the absence of an approval capability inside the component that generated the finding. Severity also changes the operational expectation. A critical finding is due in four hours; a low-severity item can wait seven days. But reviewing the implementation forced me to narrow my own claim. The current API exposes approval and rejection endpoints without an authentication dependency. Reviewer identity is supplied as a string in the request body, with “auditor” as its default value. The orchestrator also marks an audit run COMPLETE after placing its findings in the queue, even though those findings remain pending. The prototype therefore enforces a workflow boundary inside the orchestrator. It does not yet prove that the caller approving the finding was an authorized human. That distinction matters in a regulated system: “The agent cannot approve its own output” is weaker than “only a separately authenticated reviewer with the correct role can approve it.” Before calling this production-ready, I would add identity-aware authentication, role-based authorization, separation-of-duties checks, reviewer identity derived from an access token, transactional state transitions, and an explicit WAITING_FOR_REVIEW run state. The architecture had the right instinct. The security boundary was incomplete. The ledger is tamper-evident, not tamper-proof The second mechanism is a SHA-256 hash-chained audit ledger. Each finding event — drafted, approved, or rejected — stores a hash derived from the previous ledger entry and the current event: chain_hash = sha256(previous_hash + finding_hash) If someone edits an old entry without updating the rest of the chain, verification fails at that point. This is better than a plain mutable log because partial historical edits become detectable. It is not a blockchain, and it is not a complete answer to privileged tampering. A user with permission to rewrite the entire JSON ledger can modify history and recompute every later hash. The chain will verify because the attacker replaced both the data and the evidence used to verify it. Hash chaining proves internal consistency. By itself, it does not prove that the chain is the original one. For a defensible production design, I would anchor periodic chain heads outside the application’s write boundary. Options include signed checkpoints, a signing key held in a key-management system, append-only storage retention, or an external transparency service. That would turn the current local tamper signal into evidence that a privileged application user cannot silently recreate. This also changed the language I would use to describe the feature. Tamper-evident is accurate. Cryptographically verified is too broad unless the root used for verification is independently protected. The metric that looked best hid the real problem The headline result was 100% recall in both evaluation modes. Every injected defect was detected. That sounds excellent until precision enters the picture. The deterministic run produced 83.3% precision, 100% recall, and an F1 score of 0.909. With five injected defects and no misses, those values correspond to five true positives and one false positive. The GPT-4o-mini run also found all five defects, but precision dropped to 38.5% and F1 to 0.556. That corresponds to five true positives and eight false positives. Evaluation mode True positives False positives Recall Precision F1 Deterministic 5 1 100% 83.3% 0.909 GPT-4o-mini 5 8 100% 38.5% 0.556 Eight false alarms for five real defects is not a cosmetic issue. In an audit queue, false positives consume reviewer time, delay real findings, and teach operators to distrust the system. High recall protects against missed risk; low precision creates alert fatigue. A governance tool needs both. This also exposed an error in my first headline. I had written “100% recall, 55% precision.” The 55.6% number was F1, not precision. The actual precision was 38.5%. That is exactly the kind of flattering metric substitution an audit system should prevent — including when the person making the mistake is its builder. The gap between deterministic and LLM-based evaluation is the most important result in the project. The controlled evaluator benefited from known signatures and synthetic failure patterns. Real model outputs varied in length, phrasing, and borderline behavior. Rules that looked calibrated in a controlled environment became noisy when the generator was allowed to behave like a generator. Figure 2: Detection performance in deterministic and GPT-4o-mini evaluations. Recall remained at 100%, while precision fell from 83.3% to 38.5%. Image by author. My auditor failed its own audit ThirdLine includes a “Validate the Validator” module with minimum thresholds: F1 must be at least 0.70. Recall must be at least 0.90. The false-positive rate must not exceed 0.25. The GPT-4o-mini run passed recall at 100% and judge consistency at 97.9%. It failed F1 at 0.556 and failed the false-positive-rate limit at 61.5%. The generated model card therefore labeled the system Requires Review and stated that it did not meet its minimum governance standards for deployment. That is the result I trust most. There is another important nuance. The model card identifies PSI-based drift detection as a known limitation: output-length variance from real LLM responses can create false drift findings. But the self-audit code does not autonomously isolate PSI as the root cause. Its calibration check compares observed precision with a fixed confidence assumption of 0.85, while the PSI explanation is documented separately as a suspected limitation. The accurate claim is: The self-audit exposed overconfidence and unacceptable false positives; follow-up analysis pointed to PSI calibration as one contributor. Saying the system “diagnosed its own PSI bug” would overstate what the code does. I also added an eval-as-test gate to GitHub Actions. The workflow generates a synthetic fleet, runs the audit and meta-evaluation, and fails when F1 falls below 0.70. That is useful because evaluator quality becomes a release condition instead of a dashboard metric. It is still a narrow gate. The checked-in workflow does not exercise the live GPT-4o-mini path, so passing CI does not establish that the behavior that failed the model card is ready. What I would change next First, I would calibrate each evaluation dimension separately. A single global result hides whether the false positives originate mainly in drift detection, hallucination judging, or another evaluator. Precision-recall curves by dimension would make tuning decisions visible instead of anecdotal. Second, I would replace output-length PSI with representations closer to semantic behavior: embedding distributions, task-specific outcome features, and drift tests conditioned on prompt category. Length can be a useful signal. It is a fragile proxy for meaning. Third, I would test the reviewer boundary as aggressively as the model boundary. Can an unauthenticated caller approve a finding? Can one identity both create and approve it? Can queue state be changed without a ledger event? Governance claims should have adversarial tests, not just architecture diagrams. Finally, I would externally anchor the ledger and run the real-model evaluation as a scheduled quality gate. A deterministic test is valuable for regression control. It should not become a comforting substitute for the behavior that actually failed. The implementation, evaluation code, and generated model card are available in the ThirdLine repository . The limitation report is as important as the feature list. The broader lesson is not that AI should never audit AI. It is that a probabilistic auditor needs non-probabilistic boundaries around authority, identity, evidence, and release decisions. When your AI auditor raises a finding, what proves that the reviewer was authorized and what proves that the audit history was not rewritten afterward? 100% Recall, 38.5% Precision: What Happened When My AI Auditor Audited Itself was originally published in Towards AI on Medium, where people are continuing the conversation by highlighting and responding to this story.