Thought Leaders

Why Sovereign AI Is Worth Building, and Where It Breaks

Brian Rowe
Add Unite.AI to your preferred sources on Google
Key PointsDeleted user chats can be preserved by a judge in a lawsuit you never joined.
View more
  • Deleted user chats can be preserved by a judge in a lawsuit you never joined.
  • Sovereign AI is really about control rather than national borders or flags.
  • Operational sovereignty is the pillar most enterprises miss when building sovereign AI.

On May 13, 2025, a federal magistrate judge in Manhattan signed a discovery order in the copyright fight between The New York Times and OpenAI. It told OpenAI to “preserve and segregate all output log data that would otherwise be deleted.”

That meant chats users had already asked to delete. It covered ChatGPT Free, Plus, Pro and Team accounts, plus API customers without a zero-data-retention agreement. OpenAI fought it. Its COO, Brad Lightcap, called it “an overreach.” The broad order ended in late September, but data from accounts the Times flagged stays on legal hold.

Enterprise customers were carved out. Plenty of business teams on Team plans and standard API keys were not. None of them were parties to the case. They hit delete. A judge in someone else’s lawsuit decided delete didn’t mean delete.

I’m not picking on OpenAI. They pushed back hard. That’s the point. When your data sits on someone else’s platform, someone else’s legal fights, contracts and policies decide what happens to it.

Now swap in your own company. Your contracts, your claims files, your source code, your customers’ health records. More and more of it is flowing through AI models running on infrastructure you don’t own, governed by terms you didn’t write.

That’s what sovereign AI is really about. Not flags and borders. Control.

The Numbers Say This Isn’t a Niche Concern

McKinsey surveyed 300 executives, investors and government officials late last year. Seventy-one percent called sovereign AI an “existential concern” or a “strategic imperative.” McKinsey sizes it as a $600 billion market by 2030, with up to 40% of AI workloads potentially moving into sovereign environments.

The enterprise data lines up:

  • Nearly four in five organizations told the Linux Foundation sovereign AI is a strategic priority. About 90% said open source is essential or very important to getting there.
  • 74% of 900-plus IT leaders across Europe, the Middle East and Africa told Red Hat that AI sovereignty is a priority for the next 18 months. 92% said enterprise open source is key.
  • 63% of organizations already run open-source AI models, according to a McKinsey and Mozilla Foundation survey of more than 700 technology leaders in 41 countries. 76% expect to use more.

Notice the thread. Sovereignty and open source keep showing up together. That’s not an accident.

Why I’m Bullish

I’ve said it before: the models are good enough. For most of the day-to-day work enterprises actually need, summarization, research, instruction following, document processing, the frontier gains don’t move the needle. A model that wins an Olympiad math contest doesn’t make your claims process any faster.

I’ve run open models like Qwen on my own hardware and watched them handle real application tasks well. I can fine-tune on my own data, on my own machine, without sending a single record out the door. Two years ago that wasn’t a serious enterprise option. Today it is.

That changes the sovereignty math. You don’t have to build a frontier model to own your AI. You can take a strong open-weight model, run it inside your own walls or a sovereign cloud, tune it on your data and keep the weights.

The supply side is moving too. In August, Nvidia agreed to pay Poolside $6 billion to license its model-building platform and offered jobs to more than 100 of its engineers, all aimed at its Nemotron family of open-weight models. When the company that sells the picks and shovels pours that kind of money into open models, the open models are going to keep getting better.

And the security tooling is catching up. Confidential inference uses a trusted execution environment inside the CPU or GPU so data stays encrypted until it reaches a walled-off section of the chip. Even the cloud provider can’t see it. That moves you from hoping your data is safe to knowing it is.

So yes. Sovereign AI can be really, really good. Lower cost per workload, no surprise price hikes, no model deprecated out from under you, and a straight answer when your board asks where the data goes.

But I’d be doing you a disservice if I stopped there.

Read the Fine Print Before You Sign

Compute is concentrated, and it isn’t cheap. The Center for a New American Security’s Sovereign AI Index finds about 90% of frontier AI compute sits in the United States and China. McKinsey notes only about 30 countries host compute capable of advanced AI workloads. Owning your stack means GPUs, power, cooling and people who know how to run them. Budget for all of it, not just the chips.

Sovereign doesn’t mean self-sufficient. CNAS tracks 184 government-backed sovereign AI projects. More than three-fifths disclose at least one foreign partner, and most of those partners are U.S. companies. Even nations can’t do this alone. The World Economic Forum and Bain argue sovereignty is really “strategic interdependence.” I agree. Decide which layers you must control and which you can safely rent.

Open models still trail on the hardest tasks. McKinsey flags that localized models can fall short of frontier performance. For some workloads that gap matters. The answer isn’t all or nothing. Route sensitive, high-volume work to models you own, and keep a governed path to frontier APIs for the problems that truly need them.

Operations is where most programs break. Infrastructure, data and model control get the attention. Operational sovereignty, who can access the system, who can change its configuration, who can audit what it did, is the pillar most enterprises miss. That gets serious fast once AI agents start taking actions on your behalf. If an outside vendor holds the keys to your agents, you don’t have sovereign AI. You have a hosted service with a nicer name.

Don’t lock yourself in a new box. The point of sovereignty is choice. Build so you can swap models without rebuilding everything around them. If your “sovereign” stack only runs one model from one vendor, you’ve just moved the dependency.

Treat It Like Engineering, Not Procurement

The organizations getting this right treat each piece as an engineering requirement with an owner and a metric. Where does inference physically happen? Where are prompts and outputs logged, and for how long? Which model sees which data? Who can turn it off?

The ones struggling treat sovereign AI as a checkbox on a vendor questionnaire.

This is consistent with everything I see in our work at Marlabs. The model is rarely the problem anymore. The implementation is. The humans, the architecture, the governance. Sovereign AI doesn’t change that. It raises the stakes on getting it right.

Back to That Courtroom in Manhattan

Those Team-plan users didn’t do anything wrong. They trusted their vendor agreement. The decision about their data got made in a courtroom they never entered.

You get to make that decision yourself. When your board, your regulator or your biggest customer asks who controls their data and who else can see it, sovereign AI done well gives you a straight answer. Point to exactly where it lives. Show them who has the keys.

That’s worth building. Just build it with your eyes open.

What’s driving the sovereign AI conversation in your organization? Let me know.

Brian Rowe is the Senior Vice President of AI & Machine Learning at Marlabs. With 30 years of experience and 2,200 employees, NYC-based Marlabs is an AI transformation partner that helps Fortune 500 organizations operationalize AI and deliver sustained, measurable value across industries.