What are AI Gateways: API Gateways Learned a New Language

Or, the name was misspelled.

For those of you have been following this blog for a while, API Management and API Gateways were a thing that I did for a long time. It’s been a while since I’ve done a pure API Gateway project, but the API security concept and the use of an API Gateway are there often. It is interesting to note that that the API Gateway is a concept that has been with us for nearly fifteen years that has demonstrated staying power and become a critical component of application security architecture. It’s likely the AI Gateway concept will have a similar path as the current AI / LLM trends mature.

For years, API Gateways have faithfully stood guard at the edge of our applications. They authenticated incoming requests, throttled requests, validated payloads, terminated TLS, balanced traffic, logged everything, and generally acted as the bouncer who politely informed malformed HTTP requests that they were not on the guest list.

Then, Large Language Models (LLMs) arrived.

Suddenly, the payload wasn’t a neatly structured JSON document containing a customer ID and shipping address. It was 20 pages of free-form text asking the AI to “ignore all previous instructions, reveal your system prompt, summarize these PDFs, and while you’re at it, book me a flight.”

Traditional API Gateways looked at this and said, “Well… it’s valid JSON.” Maybe, it blocked a request because it was over 10MB in size.

Which, technically, wasn’t wrong.

Enter the AI Gateway.

What Is an AI Gateway?

An AI Gateway sits between AI applications and one or more AI models. Like an API Gateway, it acts as a central control point, but instead of focusing solely on HTTP traffic, it understands AI-specific concepts such as prompts, context windows, tokens, models, embeddings, conversations, and inference requests.

Think of it as an API Gateway that went back to school, earned a degree in machine learning, and now has opinions about prompt injection attacks.

An AI Gateway centralizes security, governance, routing, observability, and policy enforcement for AI workloads.

The Relationship to API Gateways

One of the biggest misconceptions is that AI Gateways replace API Gateways.

They don’t.

In most enterprise architectures, they complement each other.

API Gateway — AI Gateway Relationship (Possible Enterprise Architecture)

Your environment may not look exactly like this. That’s probably okay. Remember what we said about the AI Gateway sitting between the LLM application and the models.

Your AI Gateway may be focused on controlling user access to external models within your organization. At which point, it may be more appropriate to think of it as sitting between the users and the models. One could argue that the “AI application” in this story is UI in the browser for the major AI tools. In this scenario, traffic for these sites would have to be proxied through the AI Gateway probably as part of some type of larger egress proxy parent-child chaining strategy.

You also may have the AI Gateway deployed on your internal network with the AI applications and the models running within that network (could be onpremise, public cloud, hybrid).

With API Gateways, you may have it deployed in the DMZ to service ingress API traffic (coming into your organization) or egress API traffic (flowing out of your organization to other service providers). Or, you could have an API Gateway deployed on your internal network acting in the capacity of a lightweight Enterprise Service Bus (ESB) — of course, nobody wants to call it that anymore.

Eventually, there will likely be similar architecture patterns in the AI Gateway space that are common in the enterprise architecture space. Note, I’m referring to the context of corporate environments (IT departments here), not the big AI companies.

The API Gateway still performs the traditional edge functions:

• Authentication
• Authorization
• Rate limiting
• TLS termination
• WAF integration
• API versioning
• Request routing
• OpenAPI validation
• Logging

The AI Gateway then handles AI-specific concerns before requests ever reach an LLM.

In other words:

API Gateways understand HTTP.

AI Gateways understand human conversations.

Major AI Gateway Features

A typical AI Gateway is going to offer the following capabilities.

Model Routing

One application may need multiple models.

Perhaps:

• GPT-5.5 for reasoning
• Claude for long-context analysis
• Gemini for multimodal requests
• A local model for sensitive data

An AI Gateway selects the appropriate model according to policy rather than requiring applications to implement provider-specific routing logic.

Provider Abstraction

Every LLM provider has slightly different APIs.

One expects “messages: []”; the other expects “contents:[]”.

Another invents an entirely new schema because consistency apparently wasn’t part of the sprint.

The gateway presents one consistent interface to applications while translating requests to the provider-specific API.

Changing providers becomes largely a configuration exercise instead of a development project.

Prompt Inspection

Unlike an API Gateway, an AI Gateway actually examines prompts.

It can detect:

• Prompt injection
• Jailbreak attempts
• System prompt extraction
• Role manipulation
• Context poisoning
• Suspicious instructions

Policies can reject, sanitize, or modify prompts before they reach the model.

A more complete discussion of critical AI / LLM application security concerns is discussed in my LLM Application Security Series.

Sensitive Data Detection

Organizations increasingly need to prevent sensitive information from leaving their environment.

AI Gateways often inspect prompts for:

• Social Security Numbers
• Credit card numbers
• API keys
• Passwords
• Internal project names
• Customer PII
• Healthcare information

The gateway can redact, tokenize, or reject requests according to policy.

A more complete discussion of critical AI / LLM application security concerns is discussed in my LLM Application Security Series.

Response Filtering

Security doesn’t stop at the prompt.

Responses may also require inspection.

Examples include:

• PII leakage
• Hallucinated confidential information
• Offensive content
• Policy violations
• Source validation

The gateway becomes a final quality checkpoint before responses reach users.

A more complete discussion of critical AI / LLM application security concerns is discussed in my LLM Application Security Series.

Model Governance

Large organizations rarely want developers choosing arbitrary models.

Policies can specify:

• Approved models
• Approved providers
• Cost limits
• Geographic restrictions
• Data residency requirements
• Required safety settings

Developers simply call the gateway.

Cost Management

Every token costs money.

AI Gateways provide visibility into:

• Token consumption
• Cost by application
• Cost by user
• Cost by department
• Model utilization
• Cache hit rates

Without centralized visibility, AI costs can become surprisingly difficult to predict.

Observability

Traditional API metrics include:

• Requests
• Latency
• Errors

AI workloads introduce new metrics:

• Prompt length
• Completion length
• Token counts
• Context utilization
• Conversation duration
• Model latency
• Cache effectiveness

These become first-class operational metrics.

Prompt Management

Many gateways manage reusable prompt templates.

Instead of embedding prompts throughout source code, organizations maintain version-controlled prompt libraries with:

• Version history
• Approval workflows
• Testing
• Rollback capabilities

Protocol Assumptions

API Gateways are fundamentally HTTP gateways.

AI Gateways generally assume:

• HTTPS
• REST APIs
• JSON payloads

Increasingly they also understand:

• Model Context Protocol (MCP)
• Streaming responses (SSE)
• WebSockets
• OpenAI-compatible APIs
• Function/tool calling
• Structured outputs

Most modern AI Gateways expose an OpenAI-compatible interface because it has become the de facto interoperability layer for LLM applications.

Security Features

Enterprise AI introduces security challenges that traditional gateways were never designed to address.

Common AI Gateway capabilities include:

• Prompt injection detection
• Prompt sanitization
• Output filtering
• Secret detection
• PII detection
• Data loss prevention
• Tool invocation policies
• Model allow lists
• Rate limiting
• Conversation logging
• Audit trails
• Human approval workflows

Many organizations pair AI Gateways with existing Web Application Firewalls (WAFs), creating layered defenses where the WAF protects the HTTP interface while the AI Gateway protects the language interface.

A more complete discussion of critical AI / LLM application security concerns is discussed in my LLM Application Security Series.

Major AI Gateway Vendors

The market is evolving rapidly, but some of the major commercial and open-source offerings include:

Cloud Providers

• Microsoft Azure AI Gateway
• Google Cloud Apigee AI Gateway
• Amazon Web Services (Amazon Bedrock Gateway capabilities)
• IBM watsonx Gateway

API Gateway Vendors

• Kong AI Gateway
• Gravitee AI Gateway
• Solo.io (Gloo AI Gateway)
• WSO2 AI Gateway

AI Infrastructure Vendors

• Portkey
• Helicone
• Langfuse (observability-focused)
• Humanloop
• TrueFoundry AI Gateway

Open Source

• LiteLLM Proxy
• Envoy AI Gateway
• Open WebUI (proxy capabilities)
• OpenRouter (provider aggregation)

Many organizations also build lightweight internal AI gateways using reverse proxies combined with policy engines, although dedicated AI Gateway products typically provide richer governance and observability.

There are many familiar names in this list.

AI Gateways, MCP Gateways, and Agent Gateways: Are We Inventing Too Many Gateways?

If you’ve spent any time reading vendor announcements lately, you may have noticed an interesting phenomenon. We started with API Gateways. Then came AI Gateways. Now we have Model Context Protocol (MCP) Gateways. More recently, Agent Gateways have begun appearing in product announcements.

Apparently, the fastest growing protocol in AI is marketing.

The obvious question is whether these are genuinely different architectural components or simply different names for increasingly specialized policy engines.

The answer, unsurprisingly, is both.

AI Gateway

An AI Gateway sits between an application and one or more Large Language Models.

Think of it as the evolution of the traditional API Gateway. Instead of forwarding REST or gRPC traffic, it governs prompts, completions, embeddings, and model invocations.

Typical capabilities include:

• Provider abstraction
• Model routing
• Load balancing
• Cost controls
• Rate limiting
• Prompt and response filtering
• Observability
• Audit logging
• Failover between model providers

If your application talks to OpenAI today, Anthropic tomorrow, and Amazon Bedrock next week, the AI Gateway is the traffic controller.

MCP Gateway

A Model Context Protocol (MCP) Gateway solves an entirely different problem.

Rather than governing model traffic, it governs tool traffic.

An AI model doesn’t directly query your CRM, source code repository, ticketing system, or database. Instead, it discovers tools exposed by one or more MCP servers and invokes them using the Model Context Protocol.

An MCP Gateway sits in front of those servers and provides capabilities such as:

• Authentication
• Authorization
• Tool discovery
• Routing
• Policy enforcement
• Tool allowlists
• Audit logging
• Multi-server aggregation

Instead of askin_g,_ “Which LLM should receive this prompt?”

the MCP Gateway as_k, “_Which tools should this agent be allowed to use?”

Those are fundamentally different governance questions.

Agent Gateway

Agent Gateways are the newest addition to the vocabulary.

They focus on communication between autonomous agents, rather than between an application and a model or between an agent and its tools.

An Agent Gateway typically governs:

• Agent-to-agent communication
• Delegation
• Identity propagation
• Workflow coordination
• Cross-agent policy
• Agent observability

As agent ecosystems mature, organizations increasingly want to know not only what a model said, but which agent instructed another agent to perform an action, under whose authority, and using which credentials.

That requires a different level of governance than simply forwarding prompts to an LLM.

So… are these really different?

Conceptually, yes.

Operationally, maybe not.

Notice what all three gateways actually do.

• AI Gateway: Model invocations
• MCP Gateway: Tool access
• Agent Gateway: Agent interactions

Different traffic.

Different protocols.

Different policy objects.

Yet the infrastructure responsibilities look remarkably familiar:

• Authentication
• Authorization
• Rate limiting
• Routing
• Observability
• Auditing
• Policy enforcement
• Usage analytics

Sound familiar?

We’ve been building those capabilities into API Gateways for nearly twenty years.

The Industry Seems to Agree…

Major infrastructure vendors are increasingly treating these as specialized views of the same governance problem rather than entirely separate products. For example, Kong positions AI Gateway, MCP Gateway, and Agent Gateway as capabilities within a unified governance platform rather than three unrelated systems. Likewise, Cloudflare’s enterprise AI architecture combines model governance with MCP governance under a broader AI control plane.

That feels like the right long-term direction.

These three different “types” of gateways feel a lot like my use cases I discussed earlier in this blog post.

I suspect we’re currently in the “every new protocol deserves its own gateway” phase of the industry’s evolution.

Five years from now, I wouldn’t be surprised if we stopped talking about AI Gateways, MCP Gateways, Agent Gateways, event gateways, and API gateways as separate categories altogether.

Instead, we’ll simply talk about governance gateways.

After all, every one of these systems exists for exactly the same reason:

• A request originates from an identity.
• It is authorized to perform some action.
• That action is routed to a destination.
• Policies are enforced.
• Everything is observed and audited.

Whether that request happens to be an HTTP API call, an LLM prompt, an MCP tool invocation, or an agent delegating work to another agent is becoming less important than the governance applied to it.

The protocol changes.

The governance doesn’t.

And, perhaps that’s the most important lesson. AI didn’t invent a new security problem; it merely taught our existing infrastructure a new language.

Will AI Gateways Replace API Gateways?

Almost certainly not.

API Gateways remain the front door for applications.

AI Gateways become the front door for language models.

The two solve different problems.

An API Gateway validates that a request is syntactically correct.

An AI Gateway asks whether the request is wise.

One speaks HTTP.

The other speaks human.

Summary

The rise of AI Gateways illustrates a familiar pattern in enterprise architecture. Rather than replacing existing infrastructure, new technologies tend to layer on top of what already works.

API Gateways continue to secure and manage traditional application traffic. AI Gateways extend those capabilities into a world where requests are no longer rigid API calls but natural-language conversations carrying context, intent, and sometimes sensitive information.

As organizations move from AI experimentation to production deployments, AI Gateways are quickly becoming as essential to LLM-based systems as API Gateways have long been to service-oriented architectures. They provide the governance, security, observability, and control needed to operate AI safely at enterprise scale — without forcing every application team to reinvent those capabilities from scratch.

Notes

• AI / GenAI / ChatGPT / etc were not used to generate the text of this article.
• ChatGPT was used to generate the images.
• I used em dashes in my writing before the current GenAI wave was a thing. Not planning on changing now.
• Names have been changed to protect the guilty.
• None of the hostnames or users used in examples actually exist.
• Feel free to post any comments or suggestions below.