AI Is Getting Smarter; Coherence Is Getting Harder

I have spent my whole career trying to understand intelligence. I studied neuroscience because I was interested in AI and figured that learning how the brain works first was a good way to begin. Intelligence first, before “artificial” intelligence. That training left me with a habit. When a new capability or technology arrives, my first question is practical. What can this really do, and for what types of real-world problems?

As an AI guy, I rarely cheer on AI headlines. I look past the noise to understand not just the breakthroughs but also the limits of the new capabilities.

From Boring Tools to Staked Reliability

Back in 2020, when the AI safety conversation was running hot with hype, I had some plain advice. Treat AI as a tool, a powerful one, but still a tool. Stick to the boring use cases. Work at the task level, where the technology was reliable enough to help. My optimistic scenario was one of boring but useful tools.

When the agentic wave began building a couple of years later, I brought the same lens and came away unconvinced. An agentic system is only as strong as the tasks beneath it, and I had not yet seen the task-level reliability required for these systems to be effective.

However, the hype was growing. By the time I attended RSA early last year, the gap between talk and substance was still hard to ignore. There wasn’t even a shared understanding of what “agentic” meant; everyone had a different definition and perspective. And I worried about something structural. Take ten agents, each about ninety-five percent accurate. Chain them so the output of one becomes the input of the next. By the end of that chain, you are down to about sixty percent accuracy. Reliability does not survive being stacked. By then, I was starting to think about systems of agents, though my concern was still whether they could be trusted to work and make a meaningful difference.

Then, the rapid improvements in coding agents changed my mind.

Within months of my article on the topic, the incredible pace of progress made one thing clear to me. Reliability would be a matter of time for several real-world tasks. Better models with better engineering harnesses built around them were going to close the gap I had been worried about. The ceiling I had been concerned about would lift.

That is when the real problem came into focus, and it was not the one I had been watching. Once you can trust the individual agents, you don’t stop after deploying the first. You deploy many. Fleets of them, across every function, each capable, each doing its job. And a new question appeared that had nothing to do with reliability. If every agent in the fleet does what it was built to do, does the fleet still add up to what the organization intended?

The Need for Organizational Proprioception

A body with capable limbs cannot move well without proprioception, the constant inner sense of where all its parts are and what they are doing. The cerebellum does more than react to that feedback. It predicts. When the brain issues a movement command, it forms an expectation of the sensation that movement should produce, then checks that expectation against what actually shows up. The gap between the two is the signal that corrects what comes next. Skilled movement is a loop of predicting the result and correcting for the difference.

An organization operating fleets of agents needs the same sense of itself. It must know, continuously, where its systems are and how far they have drifted from their intended state. Without that inner awareness, capable parts do not combine into coordinated action. This is not a sensing mechanism you add to a body later to make it safer. You need it for the body to move at all.

There is a well-studied case of a man named Ian Waterman who lost this sense in most of his body. He learned to move again but only by watching himself, steering every step and reach with his eyes. It works. It is also exhausting and fragile. Turn off the lights, and he cannot coordinate at all. An organization that governs its agents through manual audits and periodic reviews is in his position. It compensates with constant effort for a sense it never built in, and that compensation costs more than what it replaces. The whole system collapses the moment conditions change.

There is another stranger failure worth naming, described in a fascinating book by neuroscientist V S Ramachandran. When a limb is amputated, the brain does not fall silent. It continues to generate signals for the missing limb, and the person feels them vividly. Governance can fail the same way. Strip the real judgment from an oversight function through restructuring or neglect, and the function does not go quiet. Reviews still get completed, and metrics still get produced. The organization keeps feeling the sensation of oversight while the judgment behind it has been hollowed out. Here, my 2020 advice comes back, about transparency. There is no oversight without transparency, and there is no transparency in a system that only produces the appearance of being watched.

Coherence as the Scarce Resource

Intelligence is getting cheaper. Soon it will sit in every workflow and every tool. The scarce resource in that world is coherence, the living link between what each system does on its own and what the enterprise is trying to achieve. Coherence is the connective tissue that keeps distributed intelligence aligned with one purpose.

I spent years studying how a brain keeps its many parts working as one. The agentic era asks the enterprise the same question. The goal has not changed since 2020. Boring but useful, still. Now the boring, useful thing to build is coherence itself.