Private AI Needs Infrastructure That Can Evolve With the Workload

Two years ago, almost every enterprise I talked to assumed AI meant calling a frontier model through an API. That assumption is breaking down. Open-weight models have gotten good enough for a lot of day-to-day work, and what makes an enterprise AI application valuable is usually the company’s own data, not the model behind it. Once companies see it that way, keeping the model close to their data and under their control is just common sense.

That’s what private AI is. You run the model in your own environment, you tune it on your own information, and you decide where it operates and what it’s allowed to touch. If you’re a bank or a pharma company, you were never going to be comfortable putting proprietary research or customer records in someone else’s tenant anyway. For everyone else, the push is coming from the inference bill at scale, and from not wanting to be dependent on a single provider for something that is becoming core to how the business runs. According to Linux Foundation research, 63% of companies surveyed were using an open AI model, highlighting how widely these alternatives have already been adopted.

The part people underestimate is that when you bring the model in-house, the infrastructure comes with it. Power, cooling, compute, where it lives, how it connects to everything else. Those used to be your cloud provider’s problem. Now they’re yours. Most companies have never had to make these calls before, and the technology is changing faster than anyone can pour concrete. If you size it wrong, you own capacity that doesn’t fit the workload you actually end up with. Flexibility has to be designed in up front. Retrofitting existing infrastructure, where possible, can be expensive, and some providers simply can’t do it because their facilities were never built for this kind of density.

Build the Base, Rent the Spike

Start by recognizing that you don’t need the biggest model for every job. Summarizing documents, classifying tickets, and answering questions using your own knowledge base, are all tasks that an open-weight model can run privately, at a cost you can forecast. Save the frontier models for the work that really needs them.

That tells you what shape the infrastructure should take. Build private capacity for the steady demand you can see coming every week. Rent capacity from outside for the spikes and for the heavy jobs that exceed what you’ve got. This is how companies have run compute for a long time and AI doesn’t change the logic; it just raises the density of the base and widens how far the peak can swing.

The challenge is accurately sizing the base while making the private environment and the rented one behave like one system. Both need access to the same data and operate under the same controls, without requiring companies to rebuild the pipeline every time a workload moves from one to the other.

Where Does the Infrastructure Go?

Buying the GPUs turns out to be the easy decision. Then you must put them somewhere, power them, cool them, and connect them to the rest of your estate. High-density AI hardware doesn’t drop into a standard enterprise data center without significant changes to the power and cooling systems, and it doesn’t belong in every site you run. In Uptime Institute’s 2025 AI Infrastructure Survey, 37% of respondents identified power availability as one of the main factors influencing where they run AI inference workloads.

Most companies land on a mix of colocation and cloud. What usually decides it is the data. It’s already spread across several places, it’s expensive to move, and in regulated industries, some of it can’t leave a country or a particular environment. Consolidating everything into one building next to the GPUs sounds clean and almost never happens.

So the question isn’t only where the compute sits. It’s how the compute reaches the data it needs, securely, without piling on cost and complexity you’ll regret in three years. Think of this as the third exchange. The first was the internet, and connectivity solved it. The second was cloud, and fabric solved it, enabling software-defined interconnection that allowed businesses to connect to different providers from a single location. AI is the third, and fabric alone doesn’t solve it. It needs power, a great deal of it, in the same place the fabric is. The companies that get private AI right are going to be the ones that put compute where power and connectivity meet instead of running those as two separate procurement exercises.

Connecting Private AI as Workloads Change

Agentic AI makes the connectivity question more urgent. A conventional application takes a prompt, calls a model, hands back an answer. An agent is in motion the whole time. It calls several models, queries multiple systems, pulls data from wherever the data happens to be, and hands work off to other agents. Every one of those hops crosses a boundary, introducing latency and raising the question about who is allowed to access what.

Now put the model in one location, the data in another, and a rule that some of that data must stay inside a particular country or a private environment. Companies need real control over those paths. Which systems can reach which data, what happens when a workload moves, how a new model gets wired in without disturbing everything already running.

That’s why the infrastructure must be programmable. The connections between models, data and environments should be something you configure and change, not an engineering project each time. Increasingly, it won’t be a person doing the configuring. Standards like the Model Context Protocol, or MCP, give an agent a common way to talk to the tools and systems around it. An agent that needs a connection to a new data source or a new model could request it, the same way it would call any other service, with the appropriate policy and controls remaining in place. That’s what programmable infrastructure means in practice. New models show up every few months. New applications will need paths that don’t exist today. If every change means a rebuild, you’ve lost the flexibility you went private to get.

Making the Economics Work

Getting the capacity right is a critical success factor, and it’s about more than the total number. Companies doing this well have a clear understanding of the relationship between the power they put in and the tokens they get out. Power to tokens is the unit of economics for private AI, and if you understand that for your own workloads, you can size the environment with real confidence. The other piece is headroom. New capabilities are coming to market every quarter, and the enterprises planning well always leave room for what’s next rather than building to the exact edge of today’s requirement. The best-laid plan combines the two. Know your power-to-token economics cold, then plan a sensible margin above it.

And the GPUs are only part of the bill. Housing, cooling, powering and connecting them to the rest of the business can match the hardware spend. Then there’s the people. Most enterprises have never operated a high-density compute environment, and that skill set is not something you hire in a quarter.

Which is why the base and the spike are important. Build enough private capacity to carry your normal load, rent on top when demand exceeds it, and you avoid paying for hardware that sits idle most of the year. You also keep the ability to change course as the models and the applications change, which they will.

Planning for What’s Next

Underneath all of this there’s a timing problem. A data center takes about two years to build. In two years, the models, the hardware and the applications enterprises want to run will look considerably different than they do today. Whatever you commit to now is going to be running workloads nobody can fully describe yet.

Companies are still figuring out which workloads belong in a private environment and when they need outside capacity, and that’s a reasonable place to be. The thing to avoid is making infrastructure commitments that take the decision away from you before you’ve had a chance to make it. Leave room for different models, for more capacity when you need it, and for workloads to move.

Whether private AI pays off over the long run will depend less on which model you choose and more on whether your infrastructure can keep up as your requirements get clearer. Build for the workload you have today, keep the connections programmable, and put the compute where fabric meets power.