AI Governance: Or, Congratulations on Inventing a Robot That Can Make Bad Decisions at Scale

Every new technology eventually discovers governance.

Usually, after something catches fire or 40 square blocks are leveled in the resulting explosion (which is obviously bad).

Modern Artificial Intelligence (LLM-based applications / capabilities) is no exception.

The industry has spent the last few years demonstrating that while AI is remarkably good at writing code, summarizing documents, generating cat pictures, and explaining quantum mechanics using pirate metaphors, it can also enthusiastically amplify every terrible governance decision an organization has made for the last twenty years.

The good news?

We’ve learned some valuable lessons.

The bad news?

We learned most of them the entertaining way.

The Helpful Chatbot

A large organization proudly deployed an internal AI assistant.

Employees loved it. You could ask:

• “Where’s the vacation policy?”
• “What VPN do I use?”
• “Summarize this 200-page PDF.”

Everything was wonderful.

Then someone asked_,_ “What is our CEO’s salary?”

The chatbot answered.

Someone else asked, “List everyone currently under disciplinary review.”

The chatbot answered.

Another employee requested_, “Show me every unreleased acquisition currently under NDA.”_

The chatbot…

…very helpfully answered.

Nobody had actually considered that connecting the AI to every document repository in the company might require something as old-fashioned as access control

The chatbot wasn’t hacked.

It wasn’t malicious.

It was simply the most efficient insider threat ever created.

Prompt Engineering by Chaos Theory

Another company proudly announced they had standardized on AI.

Every department was encouraged to build prompts.

Marketing had prompts.

Legal had prompts.

Finance had prompts.

Engineering had prompts.

Nobody stored them.

Nobody reviewed them.

Nobody versioned them.

Nobody documented them.

Within six months there were approximately 4,700 slightly different prompts all attempting to answer the same customer question.

Nobody knew which one was correct.

One contained confidential legal guidance.

Another accidentally instructed the model to ignore company policy.

A third confidently explained tax law using what appeared to be the screenplay from a courtroom drama.

Congratulations.

You have successfully reinvented configuration drift.

Only now it argues.

The Cost-Optimized Apocalypse

An executive discovered the AI bill.

It was…larger than expected.

A meeting was called.

Someone proudly proposed_,_ “Let’s disable all safety checks. Those use tokens.”

Another suggested_,_ “Can we stop validating inputs?”

A third asked_,_ “Do we really need output filtering?”

The finance department celebrated the projected monthly savings.

Three weeks later…

The public chatbot enthusiastically explained how to manufacture illegal substances, insulted several customers, leaked fragments of internal prompts, and recommended a competitor’s products.

The organization ultimately spent several hundred thousand dollars on incident response to save a few hundred dollars in token costs.

The quarterly report described this as “An opportunity for operational improvement.”

AI Doesn’t Replace Governance

One of the strangest (read, utterly f*****g insane) assumptions I’ve encountered is that AI somehow changes the laws of engineering.

It doesn’t.

If your identity management is poor…AI will happily expose it faster.

If your data classification is inconsistent…AI will enthusiastically consume all of it.

If your access controls are weak…AI becomes the fastest unauthorized search engine ever built.

Artificial Intelligence isn’t a replacement for governance.

It’s an amplifier.

It turns small governance mistakes into very efficient governance failures.

The Good News

Fortunately, AI governance isn’t magic.

In fact, much of it looks suspiciously like the security and architecture practices we’ve been preaching for decades.

The names are different.

The principles are not.

Know Your Data

The first question shouldn’t be_,_ “Which model should we use?”

It should be_,_ “What data are we willing to send to one?”

Not every document belongs in an external model.

Not every model belongs inside your network.

And absolutely not every spreadsheet should become training material.

If your data classification program currently consists of:

“Stuff”

and

“Really Important Stuff”

…you may have some homework.

Identity Still Matters

Every AI request should have an identity.

• Who asked?
• What application?
• What permissions?
• What tenant?
• What purpose?

If your AI gateway can’t answer those questions…

…neither can your auditors. Safety tip: that’s bad.

Put Something in the Middle

Direct application-to-model communication sounds wonderfully simple.

Until your twentieth application decides to use a different prompt.

Or a different model.

Or a different safety policy.

Or accidentally sends customer Social Security numbers to an external API.

This is why AI Gateways exist.

They provide:

• Authentication
• Authorization
• Prompt inspection
• Data Loss Prevention
• Policy enforcement
• Rate limiting
• Observability
• Audit logging

Think of them as API Gateways that went back to graduate school.

A more complete discussion of critical AI / LLM application security concerns is discussed in my LLM Application Security Series.

Version Everything

Prompts.

Policies.

Guardrails.

Evaluation datasets.

Safety configurations.

If changing a prompt can change business behavior…

It’s source code.

Treat it like source code.

Trust, But Verify

Models change.

Providers update them.

New versions appear.

Behavior shifts.

Yesterday’s correct answer may become tomorrow’s confident hallucination.

Evaluation isn’t something you perform before deployment.

It’s something you perform continuously.

The AI equivalent of “it compiled” isn’t nearly enough.

Humans Still Belong in the Loop

Egats! Why does this one even need to be stated?

There remains one remarkably effective quality assurance mechanism.

People.

The goal isn’t to remove humans from every decision.

It’s to remove humans from the boring decisions while keeping them involved in the expensive ones.

Your AI can summarize a contract.

It probably shouldn’t sign one.

It is often lost in the modern AI conversation what an IBM training manual published in 1979 stated, “A computer can never be held accountable. Therefore a computer must never make a management decision.”

Try to remember that. Otherwise, one day, some legal decision or public opinion will shove that and its implications up one end (of you) and pull it back out the other side. Most likely, you will not enjoy learning this lesson the hardway.

Governance Should Enable, Not Prevent

Good governance helps people innovate safely.

Bad governance creates seventeen approval committees before anyone can test a prompt.

One produces trustworthy AI.

The other produces PowerPoint.

If your governance process takes longer than building the AI application itself…

…you may have accidentally optimized for meetings.

In the meantime, you can start introducing yourself as Mordak The Governator (be it ever so impressive). Sarcasm.

Final Thoughts

AI governance isn’t really about AI.

It’s about governance.

Organizations are discovering that AI governance feels remarkably familiar, if they already understand:

• Identity
• Least privilege
• Data classification
• Change management
• Logging
• Risk management
• Security architecture

Organizations that ignored those disciplines are discovering something else.

Artificial Intelligence doesn’t create governance problems.

It simply finds the ones you’ve been carefully ignoring and demonstrates them at approximately 10,000 requests per minute.

So, before asking your shiny new chatbot to revolutionize the business, take a moment to ask a much older question_,_ “Who should be allowed to do what, with which data, under what controls, and how will we know when something goes spectacularly wrong?”

It isn’t a particularly glamorous question, but unlike your latest large language model, it’s one that has never hallucinated.

Notes

• AI / GenAI / ChatGPT / etc were not used to generate the text of this article.
• ChatGPT was used to generate the images.
• I used em dashes in my writing before the current GenAI wave was a thing. Not planning on changing now.
• Names have been changed to protect the guilty.
• None of the hostnames or users used in examples actually exist.
• Feel free to post any comments or suggestions below.