Why Procurement Is Becoming AI’s Most Powerful Regulatory Tool

For all the attention being paid to AI legislation, some of the most consequential rules governing the technology may end up coming from a truly unexpected place: procurement departments.

Like many regulatory beginnings in the United States, California offers an early example: the state has built specific requirements for buying products and services that use generative AI. State entities must assess the risk associated with GenAI purchases, with moderate- and high-risk uses receiving additional scrutiny from the California Department of Technology (CDT). Solicitations require GenAI disclosure language, and vendors may have to disclose their use of the technology even when AI is incidental to the product or service being purchased.

Even more interesting is what happens after the deal is done. California’s GenAI procurement guidance extends oversight beyond the initial purchase. Contract managers are expected to monitor GenAI technology over time because certain changes to the technology can trigger another risk assessment.

This changes things for vendors since getting through procurement is no longer a matter of answering a security questionnaire and handing over some documentation. AI governance is starting to become part of the price of admission for doing business in California. And while this is the current situation in California, it often leads other states when there is not federal legislation in place, meaning that companies would be shortsighted to view this as a California procurement problem.

When the Buyer Starts Making the Rules

How did procurement departments become ground-zero for AI governance? Governments can spend years debating legislation, but procurement rules can be nimbler because they have a much more immediate audience: anyone who wants the contract.

Without comprehensive AI legislation, buyers have found another way to establish their own expectations around AI. They can decide what they need to know about a vendor’s use of the technology, what evidence they expect to see, and what level of risk they are willing to accept before signing a contract. For large buyers, particularly governments and major enterprises, those requirements can carry considerable weight.

Private-sector procurement teams have their own reasons to start asking questions around things like where AI is being used or what evidence exists that there are appropriate controls in place. Enterprises already spend considerable time examining the security and privacy practices of their suppliers. The introduction of AI adds another set of risks that buyers increasingly need to understand before handing a vendor their data, putting its technology into a critical process or allowing it to make decisions that affect customers and employees.

Your AI Risk May Belong to Somebody Else

One of the more important things about California’s approach is that it recognizes how difficult it has become to draw a clean line around an organization’s use of AI. Whether it has purchased its own AI offering, works with consulting firms delivering  services, or simply uses cloud-based services, AI is embedded within organizations or their supply chains.

California’s procurement process accounts for some of this. It covers IT and non-IT purchases involving GenAI, including consultants using the technology, and calls for risk assessment when GenAI is identified during procurement. 

The process is similar to when cybersecurity teams learned that protecting their own systems wasn’t enough if a supplier with access to their data or infrastructure had weak controls. That realization turned third-party cybersecurity into a major part of vendor risk management, a direction that AI seems to also be heading. 

It also creates an awkward problem for organizations that have built separate programs for security, privacy, AI governance and every new compliance obligation that comes along. The risks don’t respect those organizational boundaries. An AI application can create privacy questions, security exposure and regulatory obligations at the same time.

Building another silo to govern it only adds another place for something to fall between the cracks.

The Trouble With Proving Compliance Once

An additional complication to an already complex issue is that AI moves. Models are updated, vendors roll out features, employees find uses for tools that nobody anticipated when they were purchased, and data changes. This fluidity means that regulation needs to change, too.

Yet a lot of compliance activity is still built around a moment in time. An audit is coming, so evidence is gathered. A customer sends a questionnaire, so somebody tracks down the answers. A new requirement arrives, and another compliance project begins.

This approach was already cumbersome, but with AI, it becomes increasingly difficult to defend. A risk assessment performed when a product was purchased may be perfectly accurate at the time, but be badly out of date a year later.

California appears to recognize this, as its contract monitoring guidance calls for ongoing evaluation of GenAI deliverables, including inaccurate output, fabricated content and bias. Changes or additions to GenAI technology in contracts that previously required CDT consultation can also require reassessment.

The NIST AI Risk Management Framework takes a similar view of AI risk management, treating it as an ongoing process across the AI lifecycle. Doing so will require a different mindset from organizations accustomed to gearing up for the next audit and then breathing a sigh of relief when it is over.

Procurement Wants Evidence, Not a Scramble

There has been plenty of excitement around automated compliance, particularly as companies try to keep up with a growing number of security, privacy and AI requirements. The attraction is understandable, but the danger comes when efficiency gets confused with governance.

Completing a workflow doesn’t tell you whether the underlying risk has changed. For example, a vendor that wasn’t using AI for a particular service six months ago may be using it now. This is why compliance needs to function more like a loop than a finish line. Organizations need to know what they are governing, check that controls continue to do what they are supposed to do, and fold new risks and requirements into the process as they appear.

Done properly, that also makes procurement considerably less painful. When a customer asks how AI is governed, the answer isn’t sitting in a document someone wrote last year for an audit. There is a current, ongoing body of evidence showing how risks are being managed in practice, which is becoming commercially important.

California is showing how procurement can turn AI governance principles into requirements that vendors actually have to meet if they want the business. Other public-sector buyers can do the same, while enterprise procurement teams have plenty of incentive to ask harder questions of their own suppliers.

Companies may eventually find that AI governance isn’t something they do simply because a regulator requires it. They will govern AI because a procurement team decided to ask them to prove it, which they will happily do in order to win the business.