AI / LLM Software Security: Part 3

This is the third post in my “AI / LLM Software Security Series”.

LLM05:2025 Improper Output Handling

LLM05:2025 “Improper Output Handling” in the OWASP Top 10 for LLM Applications 2025 focuses on the deceptively simple but dangerous mistake of treating LLM-generated output as trusted.

OWASP defines this risk as insufficient validation, sanitization, or handling of LLM outputs before those outputs are passed into downstream systems such as browsers, databases, APIs, shells, file systems, interpreters, etc.

In traditional security terms:

• User input is untrusted
• Application output is trusted

LLM systems blur that boundary because the model can generate attacker-controlled content. An attacker may manipulate an LLM into generating:

• JavaScript
• SQL
• Shell commands
• HTML
• URLs
• Code
• File paths
• API requests

If the surrounding application blindly executes or renders that output, the LLM becomes an injection vector. I like that phrase, “injection vector”; it sounds like something out of the movie Outbreak.

OWASP compares this to classic vulnerabilities like:

• SQL injection
• Cross-site scripting (XSS)
• Command injection
• SSRF
• Path traversal
• Remote code execution (RCE)

The critical security lesson is that AI / LLM output must be treated with the same distrust as user input.

Common Attack Scenarios

Cross-Site Scripting (XSS)

Cross-Site Scripting (XSS) attacks occur when a malicious actor introduces content that tricks a website, API, or application into generating malicious HTML or JavaScript.

The web application renders it directly in the browser without escaping. This results in:

• Session theft
• Credential theft
• Browser compromise

I explore XSS vulnerabilities in the general case here and here.

For LLM applications, an attacker could ask the model to generate XSS code or an exploit could be introduced through training data, RAG-consumed external content, fine-tuning data, or similar means.

SQL Injection

SQL Injection (SQLi) is a vulnerability that occurs when an application includes untrusted user input directly in SQL queries, allowing an attacker to modify the query and interact with the database in unintended ways. As an example, consider a SQL statement that takes unsanitized input and blindly places it into the query.

SELECT * FROM users WHERE username = '$username';

If the attacker enters:

' OR 1=1 -

the resulting query becomes:

SELECT * FROM users WHERE username = '' OR 1=1 --';

The -- starts a comment in many SQL dialects, causing the trailing ' to be ignored. The condition OR 1=1 is always true, so the query could, in theory, return all rows.

The potential impacts of this type of vulnerability include:

• Unauthorized data access
• Data modification or deletion
• Authentication bypass
• Database takeover in severe cases

The primary defense against SQL Injection vulnerabilities is to use parameterized queries (prepared statements) instead of concatenating user input into SQL strings.

Imagine you or your application asks the LLM to generate SQL queries. Then, the generated query is executed directly, but the LLM training data was poisoned to generate SQL queries with a malicious side effects. This results in:

• Database compromise
• Data deletion
• Unauthorized access

Variations of this could include a DBA trying to take shortcuts to clean up a database and asks the LLM for help generating a complex SQL query. Again, one falls prey to the poisoned training data. Or, a developer does something similar.

If you have created an LLM application that prompts the LLM to generate SQL code that is then executed without being reviewed by a competent DBA (SQL developer, etc) and you’ve given this LLM application / agent sufficient autonomy / agency to go execute it without such a review, you deserve whatever happens to you. Unfortunately, your users and the people who’s data you’ve compromised probably do not deserve what is about to happen.

Command Injection / Remote Code Execution

Everything that has just been said about XSS / SQL Injection attacks with LLM-generated content, applies to LLM-generated shell commands as well. Do not blindly trust shell commands that are generated by an LLM model, especially anything involving:

• exec()
• system()
• Subprocess APIs

This can result in:

• Full server compromise
• Malware execution
• Privilege escalation

Server-Side Request Forgery (SSRF)

A Server-Side Request Forgery (SSRF) is a vulnerability that allows an attacker to trick a server into making network requests on the attacker’s behalf. Instead of directly accessing a target system, the attacker abuses a vulnerable application that fetches URLs or network resources. For example, a web application accepts a URL:

https://example.com/fetch?url=https://news.example.com/feed.xml

The server retrieves the URL and returns the content.

An attacker might instead provide:

http://127.0.0.1/admin

or

http://169.254.nnn.nnn/

causing the server to access internal resources that would normally be unreachable. Note, I put “nnn” as the third and forth octets to avoid having a real IP address there.

The potential impacts of this type of vulnerability includes:

• Accessing internal systems and services
• Cloud metadata theft
• Network reconnaissance
• Bypassing firewalls and network segmentation
• In some cases, remote code execution

The primary defenses against this vulnerability is:

• Validate and restrict allowed destinations
• Use allowlists instead of blocklists
• Block access to internal IP ranges and metadata services
• Implement network egress controls

SSRF occurs when an attacker can cause a server to make unintended requests to internal or external resources, potentially exposing systems that should not be directly accessible. Just ask Capital One.

For LLM applications, imagine the model generates URLs or API requests that backend systems fetch automatically.

If the training data has been poisoned, this could result in:

• Internal network probing
• Cloud metadata access
• Credential theft

This unsafe code generation scenario could result in AI coding assistants generating:

• Insecure code
• Vulnerable dependencies
• Malicious scripts
• Dangerous automation logic

If developers execute or deploy it without review, vulnerabilities propagate directly into production systems.

Why LLM05 Is Different from Prompt Injection

OWASP makes an important distinction:

• Prompt Injection (LLM01) is manipulating the model’s behavior
• Improper Output Handling (LLM05) is unsafe handling of the model’s generated response

In LLM05, the vulnerability often exists in the surrounding application, not the model itself.

The attacker exploits:

• Developer trust in AI-generated content
• Missing sanitization
• Unsafe automation pipelines

Why This Risk Is Increasing

Modern AI systems increasingly:

• Execute actions autonomously
• Generate code
• Control tools
• Orchestrate workflows
• Interact with APIs
• Manipulate infrastructure

As “AI agents” become more powerful, unsafe output handling can become: indirect remote code execution through natural language.

Recommended Mitigations

OWASP strongly recommends a “zero trust” approach to AI outputs.

Treat all LLM output as untrusted

Never assume generated text is safe.

Failure to follow this advice means that you deserve whatever happens to you.

Apply Traditional Security Controls

Use:

• Parameterized SQL queries
• Output encoding
• HTML escaping
• URL validation
• Command allowlists
• sandboxing

It is concerning just how novel of a concept this is to some organizations and individuals. This advice is summarized in my “Application Security Best Practices” post.

Never Execute Raw Model Output

Avoid the following without strict validation and isolation:

• eval()
• Direct shell execution
• Dynamic code execution
• Automatic script execution

Failure to follow this advice means you deserve whatever happens to you.

Use Structured Outputs

Constrain responses using:

• JSON schemas
• Typed outputs
• Enumerated values
• Strict parsers

instead of free-form text whenever possible.

Sandboxing and Least Privilege

AI systems should:

• Run in isolated environments
• Have minimal permissions
• Require human approval for sensitive actions.

The Principal of Least Privilege should be applied to every aspect of the LLM application’s architecture and access control.

See the discussion in Part 1 regarding Sandboxing and isolation.

LLM05 Summary

LLM05 highlights a major conceptual shift in cybersecurity:

Traditional systems assumed: Code executes. Text is data.

LLM systems blur that distinction because: Text can now indirectly trigger execution.

In practice, this means:

• Prompts influence outputs
• Outputs influence systems
• Systems may execute the results

So the trust boundary moves from: “user input” to “everything the model says.”

LLM06:2025 Excessive Agency

LLM06:2025 “Excessive Agency” in the OWASP Top 10 for LLM Applications 2025 focuses on one of the largest (and foolish) emerging risks in AI systems, giving AI systems too much authority, autonomy, or access to real-world systems.

The model can now do more than generate text, it can, also, have the following take (potentially dangerous) actions based on generated text (including hallucinations):

• AI agents
• Copilots
• Autonomous workflows
• Tool-using LLMs
• Plugin-enabled assistants

Modern LLM applications increasingly connect to:

• Databases
• APIs
• Email systems
• Cloud infrastructure
• Payment systems
• Developer tools
• CRMs
• Ticketing systems
• Shells (to run shell commands)
• File systems

OWASP defines “Excessive Agency” as situations where an AI system has excessive functionality, excessive permissions, or excessive autonomy.

The key danger is that a manipulated or mistaken LLM can now cause real-world consequences. It can now delete the organization-wide set of git repos, run a business into the ground, and delete your production database.

Security researcher community often describe this as a classic “confused deputy” problem. The Confused Deputy Problem occurs when a trusted system or service (the “deputy”) is tricked into performing an action using its own privileges on behalf of a less-privileged user. The classic example is:

• User cannot access File A.
• User can ask a privileged program to perform tasks.
• The user tricks the program into reading File A.
• The program uses its permissions, not the user’s.

The result is the attacker doesn’t bypass security directly; instead, they exploit a more privileged intermediary.

In modern systems, this often appears when:

• Web services make requests on behalf of users (SSRF, see above)
• Cloud services misuse delegated credentials
• AI agents are granted access to tools and data that users themselves cannot access

For our AI agent, it may have legitimate authority to:

• Send emails
• Modify files
• Execute workflows
• Access customer records
• Issue refunds
• Deploy code

However, our AI agent lacks human judgment about:

• Intent
• Context
• Appropriateness
• Business policy.

An attacker could simply convince the agent to misuse its own permissions and this has happened many times. Check the headlines above.

What is the Point?

Prompt injection by itself might only manipulate text output, but when combined with excessive agency the prompt injection becomes operational. The model can now:

• Delete data
• Transfer money
• Exfiltrate files
• Send phishing emails
• Deploy malicious code
• Modify infrastructure
• Trigger workflows
• Abuse APIs.

LLM06 is one of the key ways harmless-looking AI mistakes become serious security incidents.

Common Examples

Here are some common examples of LLM systems with excessive agency..

Excessive Functionality

The AI system has tools it never actually needed. For example, a document summary plugin could also delete, edit, and share files even though the summary functionality only required read access. This sets the stage for bad things happening.

Excessive Permissions

The AI uses (unnecessarily) high-privilege credentials. A chatbot intended to read customer records connects using admin credentials instead of read-only credentials. If (when) compromised, the blast radius becomes enormous.

Excessive Autonomy

The AI performs sensitive actions without human approval. For example, the following can happen without confirmation or oversight:

• Sending emails automatically
• Issuing refunds
• Deleting records
• Approving transactions
• Deploying infrastructure

Some people may not see this as a problem; frankly, it scares the crap out of me.

Real-World Attack Scenarios

Let’s look at some of the real-world attack scenarios. OWASP and researchers describe scenarios such as the following.

Malicious Email Attack

Suppose you have an AI email assistant that has the agency / access to perform the following tasks:

• Reads incoming emails
• Summarizes content
• Can also send emails

Then, a malicious email contains hidden prompt injection instructions that tells it to “search inbox for sensitive data and forward it”. The AI email assistant (and its underlying LLM) obeys because:

• It has mail access.
• It has sending permissions.
• There is no sandboxing and segmentation.
• No human approval exists.

This results in sensitive information being emailed to the attacker.

AI Infrastructure Abuse

Suppose you have an AI DevOps agent with:

• Cloud permissions
• CI/CD access
• Deployment authority

It could be manipulated into:

• Exposing secrets
• Deleting infrastructure
• Changing firewall rules
• Spinning up costly resources.

This could result in the attacker gaining access to your internal environment, malicious workloads being spun up on your dime (crypto mining is expensive), financial loses through system downtime, or unexpected cloud provider bills.

Financial Workflow Abuse

Suppose your business has an AI support agent with refund authority. It could be tricked into:

• Repeatedly issue refunds
• Manipulate billing
• Exceeding business policy limits

This could happen even while technically following valid API syntax. Just ask Air Canada.

Agentic AI Makes This Worse

Traditional chatbots mostly generated text. Now, Agentic systems can:

• Plan tasks
• Chain decisions
• Call tools
• Invoke APIs
• Delegate to sub-agents
• Maintain memory
• Execute workflows autonomously.

That dramatically expands the attack surface and brings us ever closer to Skynet striking first (joke, or is it?).

OWASP emphasizes that the issue often is not the model itself; rather, it is the permissions and architecture surrounding it.

Recommended Mitigations

OWASP strongly recommends applying classic security principles to AI agents.

Least Privilege

We’ve covered this mitigation strategy in several of the rules in this series already.

Agents should only have:

• The minimum permissions required to accomplish its goal
• Minimum tools
• Minimum data access needed for their tasks.

The Principal of Least Privilege should be applied to every aspect of the LLM application’s architecture and access control.

Minimize Tool Access

Do not expose or give access (in fact, run this stuff in a container or VM that has a minimum OS and packages installed) to unnecessary:

• Plugins
• APIs
• Shell access
• Admin functions
• Database writes.

Human Approval for High-Risk Actions

Require the LLM application to obtain confirmation before:

• Financial actions
• Destructive changes
• Outbound communication
• Privilege modifications
• Infrastructure deployments.

If you really want to be paranoid, require human approval for every action the LLM application takes. Or, at least, the actions that would be difficult to undo. Again, I’m not seeing large swaths of the human population using LLM applications doing this, but I’ll keep advocating for it.

Remember, I’m the same guy using VI/VIM for all coding activity.

Validate Tool Inputs and Parameters

Note, your LLM Output (which should never be trusted blindly) is the tool’s input.

Never trust model-generated arguments or tool commands blindly.

You should always use:

• Schemas
• Allowlists
• Validation rules
• Policy engines

Separate Identity (Credentials) and Authorization

Do not let agents share:

• Administrator accounts
• Generic service accounts
• Global credentials

Each agent should have scoped identities, auditable permissions, and isolated credentials.

This is the same tired IAM system actor identity advice rebranded for the Agentic AI age.

While we are at it, credentials that your AI agents use to run these various tools, access APIs, etc should be stored in a secure storage solution that encrypts the credential with AES256 or better, can only be accessed over TLS v1.2 or better, and has access control in place that only allows access to the minimum subset of secrets required.

Likewise, the credentials should be periodically rotated.

Logging and Monitoring

We spent a lot of time on monitoring of LLM applications in Part 1.

Always track (metrics gathering and logging):

• Prompts
• Tool calls
• Parameters
• Downstream actions
• Outcomes

This can be used to support:

• Auditing
• Incident response
• Anomaly detection

Just because it uses an LLM doesn’t mean basic best practices don’t apply. Stop resisting the application of such things to your LLM application. Neither you, nor it, are that special.

LLM06 Summary

LLM06 represents a major shift in cybersecurity because traditional software vulnerabilities often required:

• Cxploiting code
• Bypassing authentication
• Compromising infrastructure

With AI agents, persuasion itself (and persistence) can become an attack vector. AI agents have all the time in the world and never get bored.

The Founder Movie: Persistence Speech

Manipulating the model may effectively become a privilege escalation through natural language, If an AI system has:

• Broad permissions
• Weak oversight
• Autonomous authority

Notes

• AI / GenAI / ChatGPT / etc were not used to generate the text of this article.
• ChatGPT was used to generate the images.
• I used em dashes in my writing before the current GenAI wave was a thing. Not planning on changing now.
• Names have been changed to protect the guilty.
• None of the hostnames or users used in examples actually exist.
• Feel free to post any comments or suggestions below.