Governed Is Not Verified: The Missing Layer in Enterprise AI

AI governance can establish accountability, document controls and authorise deployment. It cannot prove that a generative AI system will preserve the facts entrusted to it. Enterprises now need a second layer: continuous, evidence-based verification of AI outputs.

The Compliance Illusion

Hong Kong offers a useful example of how AI governance is maturing. Its Digital Policy Office has established an Ethical Artificial Intelligence Framework for government AI projects, supported by the Hong Kong Generative Artificial Intelligence Technical and Application Guideline. The framework includes an AI Application Impact Assessment that project teams are expected to begin during planning and maintain throughout the application lifecycle.

The assessment is not merely a final form completed before launch. It examines the purpose of the system, accountability, data use, stakeholder impact, human oversight, privacy, security, fairness, model performance and operational monitoring. Risk-gating questions identify applications that require endorsement from the IT Board or Chief Information Officer. Other projects remain subject to normal project-governance approval. The Hong Kong Judiciary separately requires judges, judicial officers and support staff to understand the capabilities, limitations and risks of generative AI while retaining responsibility for their work.

This is a sensible approach. It creates ownership, forces risks into the open and provides a documented basis for deciding whether an AI application should proceed. Similar ideas appear in the NIST AI Risk Management Framework, ISO/IEC 23894, the EU AI Act and sector-specific guidance across financial services, healthcare and critical infrastructure.

But there is a problem.

An AI system can satisfy its governance requirements and still produce a materially false answer. It can pass privacy review, operate inside an approved environment, maintain an audit trail and keep a human nominally in the loop—while silently changing the meaning of the information it processes.

The application may be governed. Its output has not necessarily been verified.

What an Impact Assessment Actually Proves

An AI Impact Assessment can demonstrate that an organisation has:

identified foreseeable risks;
assigned accountable owners;
selected proportionate controls;
evaluated privacy, fairness, security and human impact;
documented approval and escalation processes;
planned testing, monitoring and incident response.

These are essential elements of responsible deployment. They provide evidence of due diligence and make it possible to challenge poorly conceived or excessively risky applications before they cause harm.

What the assessment cannot do is certify every future output as correct. It assesses the application, its intended use and its controls. It does not inspect every statement generated after deployment.

This distinction becomes critical when AI is used to summarise case files, extract obligations from contracts, classify incidents, prepare regulatory reports or brief decision-makers. A fluent response may preserve most of the original information while altering one date, omitting one exception or converting a qualified observation into a definitive conclusion. In a high-consequence workflow, that small semantic change may be the part that matters most.

The problem is better described as a semantic integrity failure: an AI output changes, omits, invents or misrepresents information in a way that materially alters the meaning of its source.

Examples include:

changing an amount, date, name or case number;
omitting an exclusion, warning or legal qualification;
presenting an estimate as a confirmed fact;
combining facts from different records into one false conclusion;
attaching a real citation to a claim the source does not support;
resolving conflicting evidence without disclosing the conflict;
producing a technically accurate summary that leaves out the decisive fact.

This is not necessarily deliberate manipulation. A language model has no stable intention behind the output in the human sense. It generates a statistically plausible continuation from its instructions and context. Yet from the organisation’s perspective, the operational effect can be indistinguishable from manipulation: the information presented to the user no longer faithfully represents the authoritative record.

Why Familiar Safeguards Are Not Enough

Many enterprise AI controls are useful, but they are routinely credited with capabilities they do not possess.

Human-in-the-loop

Human review is important, especially where an output affects rights, safety, money or access to services. But the phrase often hides an untested assumption: that the reviewer has the time, expertise, source access and authority required to detect an error.

A person approving hundreds of fluent AI summaries may gradually become a confirmation layer rather than an independent reviewer. Human oversight only works when the review task is defined, material claims are visible, evidence is accessible and reviewers can reject or escalate the output without operational pressure.

Retrieval-Augmented Generation

Retrieval-Augmented Generation, or RAG, grounds a model in selected external information. It is usually more reliable than asking the model to answer entirely from its trained parameters. But retrieval is not proof.

The system can retrieve the wrong document, miss a decisive passage, split context across chunks, rely on an outdated version or ignore evidence that conflicts with the emerging answer. It can also produce a correct citation beside a claim that the cited passage does not entail.

Current research increasingly evaluates retrieval, factuality and reasoning separately because an end-to-end accuracy score can conceal the component that failed. The 2025 FRAMES benchmark, for example, tests factual responses, retrieval and multi-step reasoning in integrated RAG scenarios rather than treating grounding as a single capability.

Temperature zero

Setting a model’s temperature to zero usually reduces variation. It does not make the system truthful, formally deterministic or repeatable across provider updates and infrastructure changes. More importantly, a system that consistently returns the same incorrect answer is not reliable.

Temperature is a sampling control, not a factuality control.

Structured output

JSON schemas, constrained decoding, Pydantic and similar mechanisms can ensure that the output contains the required fields and valid data types. They cannot establish that the values placed in those fields are correct.

A schema can require a decision_date. It cannot prove that the date came from the right decision.

A second model

Passing an answer to another language model can identify some errors, contradictions and omissions. It is useful as an additional probabilistic test. It is not independent proof.

The generator and reviewer may share training data, architectural weaknesses and a preference for plausible language. Both may accept the same misleading context. A model reviewing another model should therefore be treated as one signal in a validation system—not as an authority.

Audit logs

Logging prompts, retrieved material and responses is necessary for investigation and accountability. A log records what happened. It does not prevent an incorrect output from reaching a user, nor does it reveal an error unless somebody or something evaluates the logged content.

Provenance Is Not Truth

The deployment of Google’s SynthID image verification in Gemini illustrates another important distinction. SynthID embeds an imperceptible watermark in supported AI-generated or AI-edited media. Gemini can inspect an image for that signal and provide information about whether it was created or modified using Google AI.

This is valuable provenance technology. It is not a factual verification system.

SynthID may help answer, “Was this image generated or edited by a supported Google AI system?” It cannot answer, “Is what this image depicts true?” Likewise, C2PA Content Credentials can record origin and editing history, while cryptographic hashes can establish file identity and reveal later alteration. Neither mechanism proves that the original content was accurate.

The distinction is fundamental:

Control	What it can establish	What it cannot establish
Cryptographic hash	File identity and tamper evidence	Truth or semantic accuracy
SynthID	AI provenance within supported systems	Factual correctness
C2PA credentials	Recorded origin and edit history	Whether the content is true
RAG citation	A source was retrieved or referenced	That the source supports the claim
JSON schema	Required structure and data types	Accuracy of the values
Second-model review	Another model detected or missed an issue	Independent verification
Human approval	A person accepted the output and responsibility	Absence of error

Provenance, governance and verification solve different problems. Treating them as interchangeable produces a dangerous sense of assurance.

What Current Research Can Deliver

There is no generally accepted method that guarantees the reliability of unrestricted generative AI. The direction of current research is instead toward bounded assurance: defining a particular task, measuring its failure modes, restricting the model’s authority and verifying important claims with evidence appropriate to the domain.

NIST’s Generative AI Profile recommends repeated testing, evaluation, validation and verification throughout the lifecycle. It treats pre-deployment evaluation as a way to measure performance, capabilities, limits, risks and impacts—not as a universal certificate of correctness.

Several technical directions are improving the available evidence.

Claim-level verification

Instead of rating a complete answer as correct or incorrect, the system decomposes it into atomic claims. Each material claim is then connected to supporting evidence and evaluated independently. This makes a changed number, unsupported conclusion or missing qualification easier to detect than it would be inside a polished paragraph.

Retrieval-augmented correction

Retrieval Augmented Correction and related approaches examine generated claims, retrieve additional evidence and revise unsupported statements. Research published at EMNLP 2025 demonstrated that post-generation correction can improve factuality without requiring full model retraining. The result is promising, but it remains dependent on retrieval coverage and the quality of the verifier.

Adaptive verification

Not every answer requires the same degree of scrutiny. Emerging systems estimate uncertainty, disagreement or reasoning consistency and route higher-risk claims into deeper retrieval, additional tools or human review. This can make verification economically viable, provided model confidence is not mistaken for factual confidence.

Inference-time verification

Reasoning systems can generate multiple candidates, criticise them and spend additional computation selecting or revising an answer. This works especially well where the result can be tested against formal constraints, executable code or mathematical rules. It is less conclusive when one language model is asked to judge another model’s ambiguous natural-language claim.

Statistical evaluation

Reliability claims depend on the quality of the evaluation itself. NIST’s 2026 work on statistical models for AI benchmarking emphasises the need to specify the underlying statistical model and account for variation in systems, prompts, test items and evaluators. A vendor’s single “accuracy” percentage is not sufficient evidence for an enterprise deployment.

The common direction is clear: reliability does not emerge from one stronger model or one universal guardrail. It emerges from a system of constrained generation, evidence, validation and escalation.

The Missing Enterprise Layer

Enterprises need a verification architecture alongside their governance framework.

1. Protect the authoritative record

Source records should have clear ownership, version control, access restrictions, retention rules and tamper evidence. Cryptographic hashes can establish which document version was used for a particular response. An approved-source registry should distinguish authoritative material from background information and untrusted content.

2. Separate facts from language generation

Critical fields—amounts, dates, identities, account numbers, case references, control states—should be obtained from authoritative systems through deterministic queries or validated extraction wherever possible. The language model may present or explain these facts, but it should not recreate them from memory.

3. Bind material claims to evidence

Each consequential claim should carry a source identifier, source version and supporting passage or database record. Verification must test not only whether the passage exists, but whether it supports the claim and whether material context was omitted.

4. Match the verifier to the claim

Different claims require different controls:

calculations should be recomputed;
database facts should be reconciled with the system of record;
quotations should be matched against versioned source text;
business decisions should be tested against deterministic rules;
generated claims should undergo evidence-entailment checks;
unresolved interpretations should be escalated to a qualified person.

The more consequential the claim, the less its verification should depend on another unconstrained language model.

5. Make abstention a valid outcome

A system cannot be considered reliable if it is forced to answer when evidence is missing, contradictory or outside its approved scope. It must be able to return: “The available evidence is insufficient to answer this safely.”

Abstention is not a model failure. In a controlled enterprise environment, it is a safety function.

6. Test adversarially

Pre-deployment testing should cover more than obvious hallucinations. It should test outdated and conflicting documents, poisoned retrieval content, embedded prompt injection, false citations, missing qualifications, excessive permissions and changes introduced by model, prompt or knowledge-base updates.

7. Monitor verification performance

Runtime monitoring should measure unsupported-claim rates, retrieval failures, human corrections, reviewer disagreement, abstention, override frequency and performance changes after updates. Logging every response is insufficient if no thresholds trigger investigation or intervention.

From Impact Assessment to AI Integrity Case

The next step in enterprise AI governance should be an AI Integrity Case: a structured body of evidence showing why a defined system can be trusted for a defined purpose within stated limits.

An integrity case would connect the governance assessment to operational evidence:

the claims the application is permitted to make;
the authoritative sources it may use;
the controls applied to each type of claim;
the tests used to measure failure;
the conditions under which the system must abstain;
the thresholds that trigger human escalation or suspension;
the evidence required after any model, prompt, data or architecture change.

This does not promise that the AI will never be wrong. No credible framework should make that promise. It demonstrates that the organisation understands where the system can fail, limits the consequences of those failures and continuously tests whether the controls remain effective.

Governance establishes responsibility. Provenance establishes origin. Verification establishes bounded confidence in a particular output.

An enterprise AI system needs all three.

Passing an impact assessment is therefore not the end of assurance. It is the point at which the harder question begins:

What evidence do we have that this answer is faithful to the facts?

The Missing Layer in Enterprise AI