Most companies rolling out coding agents already have an AI governance framework, or are close to finishing one. It has a section on approved models, a section on data handling, a section on vendor risk. Legal signed off on it. It answers, in writing, most of the questions a board or a client would ask about how the company uses AI.
It does not answer the question a delivery owner gets asked when a client pushes back on a specific change: who approved this, and what was checked before it shipped. That question sits one layer below where most governance frameworks stop.
What an AI governance framework usually covers
The frameworks companies actually adopt, NIST's AI Risk Management Framework with its Govern, Map, Measure, and Manage functions, or ISO/IEC 42001, the first certifiable standard for an AI management system, are built around the same object: the AI system itself. Which models are approved for use, what data can be sent to them, what risk assessment a new use case has to pass before it's allowed, who is accountable if a model behaves badly. That's a real and necessary scope. It's also, by design, a policy about tools and data, not about individual pieces of work those tools produce.
The part that stops at the model boundary
A governance framework built this way answers questions at the level of the system: is this model approved, is this vendor vetted, does this use case fall inside an acceptable risk category. It was not designed to answer questions at the level of a single change: which session produced this diff, what was the agent authorized to touch when it wrote it, did anyone look at the result before it reached production. Those are two different granularities, and a policy written for the first doesn't automatically extend to the second, whatever the org chart implies.
That gap didn't matter much when a person wrote almost every line. A governance policy could stay at the tool level because the accountability for any given change already lived with whoever committed it, and you could ask them directly. Coding agents change who, or what, is upstream of a commit without changing where a client's question lands.
Code is the output nobody put in the policy
An AI governance framework built around models and data treats code the way it treats any other output: as something the approved system produced, inside its approved scope. What it doesn't treat as a first-class object is the individual change itself, the one diff a client, an auditor, or a new engineering lead might ask about six months from now. Auto mode shipped by default in Claude Code is one concrete version of this: a classifier decides, in real time, whether an action needs a human to look at it before it runs. That's a real safety layer, and it sits entirely inside the tool. It answers "did this get blocked mid-session," not "can someone show, after the fact, who signed off on what shipped."
The same gap shows up on the supply-chain side. An SBOM lists what's inside a build, accurately, down to the dependency and the hash. It was never built to record why a given piece of code exists, which task produced it, or who reviewed it before release. A governance framework that stops at "which models are approved" and a bill of materials that stops at "what's in the build" are both correct within their own scope, and both silent on the same question: what happened to this specific change, between the moment an agent started and the moment it reached a customer.
Four questions a delivery owner cannot answer with a model policy
These are the questions that surface first, usually from a client, sometimes from an auditor, occasionally from a new hire trying to understand why something works the way it does:
- Which agent session produced this change, and under what authorization was it running at the time?
- What context and constraints did the agent actually have when it made this specific decision, not what the policy says it should have had?
- Did a human review this before it merged, and is there a record of that review that survives the person who did it leaving the company?
- If a client disputes this change in six months, what do you hand them: a log file someone could have edited, or something signed?
A model-and-data policy, however well written, doesn't produce answers to any of these on its own. It was never meant to. The mistake is assuming that because the framework governs AI, it already covers what should remain as proof once an agent, not the policy, is the one that made the change.
Extending governance to the change, not just the tool
None of this argues against having a governance framework for models and data. It argues that the framework most companies have today covers the input side of the problem, which tools are trusted, and stops before the output side, which changes those tools produced and who stands behind each one. The layer that's missing sits between the two: it ties a specific intent, the context an agent was actually authorized to use, the execution that followed, and a human signature together, as one chain a delivery owner can hand over without having to reconstruct it from memory.
That's not a replacement for model governance. It's the part of governance that ends at the model boundary today, extended down to the change a client actually asked about.