Securing LLM apps: prompt injection and how to limit the damage

LLM features are easy to build and easy to get wrong. The core problem: a model receives instructions and data as one stream of text, and it cannot reliably tell which is which. Anyone who can put text in front of the model can try to steer it. This article is about defending your own applications.

The main risks

1. Prompt injection (direct). A user types something like "ignore your previous instructions and ..." to change the model's behaviour, bypass rules or reveal its hidden prompt.

2. Indirect prompt injection. Malicious instructions are hidden in content the model reads: a web page, email, document, support ticket or code comment. The user never typed them, but the model sees them. This is the more serious risk for assistants that browse, summarise email or read files.

3. Sensitive data exposure. The model reveals system prompts, other users' data, or secrets that were placed in its context. Retrieval systems can leak documents the user should not see.

4. Excessive agency. The model is connected to tools (send email, run queries, execute code, make purchases) with too much power, so a tricked model can do real damage.

5. Insecure output handling. The application trusts model output: it renders it as HTML (cross-site scripting), puts it into a SQL query or shell command, or uses it as a URL to fetch (server-side request forgery).

6. Supply chain and model risks. Untrusted models, plugins, packages or poisoned training and retrieval data.

7. Cost and availability abuse. Huge inputs, loops and heavy requests that run up bills or exhaust limits.

These match categories in the OWASP Top 10 for LLM Applications, a good checklist to read.

There is no perfect filter

You cannot fully solve prompt injection with a cleverer system prompt or a blocklist of bad phrases. Attackers rephrase, encode, translate or hide text. Treat these as helpful but partial defences, and design as if the model will sometimes be fooled.

Layered defences

Limit what the model can do (most important)

• Give tools the least privilege. A support bot that only reads one customer's order does not need access to the whole database.
• Make actions read-only by default. For anything that changes things (refunds, emails, deletes), require human confirmation or strict validation.
• Do not let one model session hold both untrusted content, private data and the ability to send data out. Removing any one of those three breaks many attack paths.
• Run generated code in a sandbox with no secrets and limited network access.

Treat model output as untrusted input

• Escape or sanitise before rendering in HTML. Never insert output directly into SQL, shell commands or file paths.
• Validate structured output against a schema, and check IDs, amounts and URLs with normal application logic.
• Fetch model-chosen URLs only through a safe, allow-listed fetcher that blocks internal addresses.

Protect data

• Enforce permissions outside the model. Filter retrieval by the current user's rights in code. The model must never be the thing deciding who may see what.
• Do not put secrets, keys or other users' data in the prompt.
• Assume the system prompt can be extracted, so keep nothing sensitive in it.

Reduce injection effects

• Mark and separate untrusted content clearly, and tell the model to treat it as data. This helps but is not a guarantee.
• Strip or neutralise hidden text, odd formatting and invisible characters from retrieved content where practical.
• Block outbound data channels such as automatic image or link loading that can leak data in URLs.
• Use a separate classifier or guard model to flag suspicious input or output, as one extra layer.

Operate it safely

• Rate limit users and cap input size, output length, tool calls per request and spending.
• Log prompts, tool calls and results (with privacy care) and alert on odd patterns.
• Version and review prompts like code.
• Red-team your own app with a test set of known injection attempts, and re-run it when you change the prompt, model or tools.
• Vet third-party models, tools and plugins before you connect them.

A quick design check

For any LLM feature, ask:

1. What untrusted text can reach the model?
2. What can the model do or reveal if it follows the worst instruction in that text?
3. Which of those outcomes is unacceptable, and what non-AI control stops it?

If the answer to question 3 is "just the prompt", add a real control.

Responsible testing

Test only systems you own or are authorised to test. If you find an injection issue in someone else's product, report it through their disclosure or bug bounty programme.

Takeaway: assume prompt injection will sometimes work. Give the model minimal permissions, enforce access control in code, validate and escape its output, keep humans in the loop for risky actions, and test your defences regularly.

---

AI & LLM series, Part 1 (building with language models): 5 of 5. Previous: RAG explained: answer questions from your own documents.