Point-in-time attestation was built for deterministic systems. Production AI isn’t deterministic.
A chatbot handling 200,000+ interactions a week cannot be assured through quarterly reviews. The conversation has to move from audit-as-an-event to audit-as-a-stream.
Continuous AI evaluation measures every production interaction of an AI system against a defined baseline, producing timestamped, sourced, versioned evidence as a byproduct of operation. Periodic attestation samples the system on a fixed cadence (quarterly, annually) and verifies controls at each sample. Continuous catches drift between samples; periodic does not.
One was built for deterministic systems and 30 years of enterprise IT. The other is what production AI actually requires.
Enterprise audit evolved around deterministic systems. Define the control. Sample periodically. Verify it works. Document the verification. Carry on. SOC 2, ISO 27001, every regulatory examination of the last 30 years was designed around this shape.
It worked for a reason. In deterministic systems, if a control passed on Tuesday, the same control passes on Wednesday. Unless somebody changed the code. And if somebody changed the code, the change is in a git log, on a change-management ticket, reviewed by a second engineer.
Periodic attestation works when between-audit drift is rare, small, and visible.
None of these moves a check-box. Each can shift behavior by the Tuesday following the change.
A foundation model vendor releases a new version. Your AI system upgrades. Nothing in your change-management system moved. Behavior is different. Quarterly attestation was designed for code changes, not weight changes.
A product manager improves a system prompt. The new prompt scores better on the narrow benchmark they tested. In production, under a slightly different distribution of user inputs, it fails in a new way. The change was small. The effect is not.
A RAG system’s knowledge base gets a monthly update. A new document contradicts an old one. The agent starts confidently citing outdated information. No engineer touched the model or the prompt; only the retrieval surface changed.
What was this AI system doing at 3:47pm on Tuesday?
If the answer is “we sampled it last quarter and it looked fine,” the answer fails. If the answer is the trace of every interaction at that moment, evaluated against the baseline, with the policy outcome captured. That is a real answer.
There is no “kind of continuous” that works. Either every interaction is measured, or there is a gap. The gap is where the question lives.
Every artifact carries a source time, not “current as of.”
Every claim points back to the underlying production trace.
The baseline used for evaluation is captured with the result.
A freshness rule per artifact. Stale evidence self-flags.
The same trace feeds the ISO, NIST, EU, AIUC views.
A one-click framework-specific audit pack, not a PDF.
Periodic audit is not obsolete. It is the right tool for the management-system question: does the organization run a sensible process for identifying, classifying, and mitigating AI risk? ISO 42001 is a management-system standard. Its periodic certification cadence is appropriate.
What is obsolete is periodic behavioral audit. An auditor who only looks at an AI system four times a year cannot claim to know how it behaves.
Compliance is expensive. Teams scramble. Evidence is assembled after the fact. Gaps discovered during the audit become remediation projects that consume the next quarter.
Today, most organizations spend 40 to 60% of their AI governance budget on evidence assembly.
Compliance is a background process. Evidence accumulates as a byproduct of production. Auditors look at the stream; the organization spends time on AI, not on audit prep.
Continuous evaluation drops the assembly cost close to zero. Budget moves to AI work that produces value.
A baseline without continuous measurement is performance art. The companion essay.
Lever 9. Agent behavior evaluation. Where continuous mode lives.
Stage 4 of the four-stage demand sequence: continuous evidence beats periodic attestation.
Audit as an event vs. audit as a stream. Continuous evidence accumulates as a byproduct of production. Not as a quarterly project.
Continuous AI evaluation measures every production interaction of an AI system against a defined baseline, producing timestamped, sourced, versioned evidence as a byproduct of operation. The output is an audit stream rather than an audit event. Auditors look at the stream; the organization spends time on AI, not on audit prep.
Three mechanisms drift AI between audits: the foundation model upgrades underneath the system, a system prompt changes in ways that feel minor, or a RAG corpus refreshes with contradictory documents. None of these move a change-management ticket. Each can shift behavior by the Tuesday following the change.
No. Periodic audit remains the right tool for management-system questions ('does your organization run a sensible AI risk process?'). ISO 42001 is correctly periodic. What is obsolete is periodic behavioral audit. Sampling AI behavior four times a year cannot describe how it behaves.
ISO 42001 requires risk monitoring; it does not specify cadence. The interpretation is converging on continuous for high-risk and high-volume AI systems, periodic management-system review for the surrounding governance program. Continuous evidence satisfies both. Periodic alone does not.
The question a quarterly audit cannot answer: 'What was this AI system doing at 3:47pm on Tuesday?' Continuous mode produces the trace of every interaction at that moment, evaluated against the baseline, with the policy outcome captured. Quarterly attestation produces 'we sampled it last quarter and it looked fine.'
Timestamped (source time, not 'current as of'), sourced (claim points to underlying trace), versioned (baseline captured with result), fresh (per-artifact freshness rule, stale evidence self-flags), cross-referenced (same trace feeds ISO/NIST/EU/AIUC views), exportable (one-click framework-specific audit pack).
