AI Oversight Has Three Blind Spots — and Most Companies Are Watching None of Them

Most of the AI safety conversation inside companies is aimed at the wrong target. Teams spend weeks arguing about which model to use, then wire that model into their email, their payments, their customer database and their code, without stopping to ask a much simpler question: when this thing does something, who is watching, and can anyone stop it?

The numbers say this is about to get expensive. Gartner reckons at least 15% of routine work decisions will be made autonomously by AI in 2028, up from close to none in 2024. In the same research it warns that more than 40% of agentic AI projects will be scrapped by the end of 2027, and one of the reasons it gives is weak risk controls. Read that again. The projects aren’t dying because the models are bad. They are dying because nobody built a way to govern them.

I build governance systems for AI for a living, so I see the same gaps over and over. There are three of them. Most companies are watching none of them properly, and plenty are watching zero.

Agents are the loud one

Everyone worries about autonomous agents, and they are right to. But most people worry about the wrong part. The unsettling thing about an agent is not that it reasons. Its that it acts. It doesn’t propose the refund; it pays it. It doesn’t draft the email; it sends it. Hand a model a set of tools and you have handed it the ability to reach into real systems and change things.

Get that wrong and the failure isn’t a clumsy sentence. Its money out the door, or a table quietly dropped from a database. And there is a sharper edge to it. Prompt injection sits right at the top of the OWASP list of risks for LLM applications, and for good reason: feed a model the wrong text and it can be talked into doing something nobody asked for. When the model can only talk, that’s a nuisance. When the model can call tools, that is an attacker holding your API keys.

The standard answer is to log everything the agent does and watch the traces. Fine. But a trace is a description of something that has already happened. It’s the CCTV footage after the till has been emptied, handy for the investigation, useless for stopping the theft.

The quiet one nobody mentions

The second blind spot never makes it onto a slide, because its dull. Its the plain API call. Not an agent, not a framework, just a piece of code somewhere in a service that assembles a prompt and fires it at a model.

Most AI in production actually looks like this, and its the least governed part of the whole stack precisely because its so ordinary. Its an HTTP request buried three layers deep in some service, written by an engineer who has never heard of your AI policy and wouldn’t know where to find it. That one call might ship a customer’s data off to a third party model, or set off an action downstream, and nothing is checking whether it should, and nothing independent is writing down that it did.

The messy one is people

The third blind spot is people, which is why it is the worst one. Your staff worked out months ago that pasting a task into ChatGPT or Claude gets it done faster, so they do it all day, usually from personal accounts you cannot see. This isn’t hypothetical. Cyberhaven’s look at real workplace usage found that a real share of employees have pasted confidential company data into ChatGPT, much of it through accounts the company has no window into.

If you want the cautionary tale, it’s Samsung. In 2023 it banned generative AI internally after engineers pasted proprietary source code into ChatGPT , three separate times in under a month. These were not bad actors. They were good engineers trying to debug faster. That is the whole trap: the leak looks identical to productivity. And your old data loss tools won’t catch it, because it’s a copy paste into a browser tab, not a file heading out of the building.

So what actually works

Line the three up and the fix stops being about agents specifically and turns into a single idea: govern what any AI does before it does it, not after. A few things that genuinely matter, learned mostly the hard way.

Sit in front of the action, not behind it. This is the whole game, and its why I keep insisting that observability and governance are not the same word. A dashboard that tells you an agent moved £40,000 yesterday is a loss report. Something that can hold that transfer for a human to look at before it leaves is a control. If your setup can only tell you what happened, you don’t have oversight. You have hindsight.

Use one checkpoint, not one per tool. Agents, API calls and staff pasting into a chatbot feel like three separate problems, so companies buy three separate tools and end up with three sets of gaps between them. The bad things live in the gaps. Everything an AI does, wherever it comes from, should have to pass the same gate.

Keep a log the AI can’t rewrite. If the system doing the thing is also the only thing recording that it did the thing, you don’t have a record. You have a diary it is allowed to edit. This is exactly where the rules are heading. The EU AI Act’s record-keeping requirement, Article 12, exists so that what a high risk system did can be reconstructed by someone other than the system. In practice that means a log kept outside the thing being logged, one that can’t be quietly altered after the fact.

Put a human on the big, irreversible calls. Not everything, do that and you will drown people in approvals until they rubber stamp the lot. But the calls you can’t take back, moving money, exporting data, deleting records, should stop and wait. The EU AI Act already makes human oversight mandatory for high-risk systems, Article 14. The catch worth remembering is that oversight only counts if the person has the time and the context to actually say no. An approval nobody reads is just theatre.

The honest gut check

None of these three is a corner case. They are the ordinary way most companies are running AI right now, quietly, without much of a net. The teams that get through the next couple of years without an ugly incident won’t be the ones with the prettiest charts. They will be the ones that decided early that everything their AI does, the agent, the API call, the pasted paragraph, goes through one gate before it happens, not after.

If you want a quick test of your own setup, answer two questions honestly. Of those three surfaces, how many can you actually see? And of the ones you can see, how many can you stop? For a lot of companies the honest answer to the second one is zero. That’s the number to fix first.