LLM Apps Are Still Apps: Bringing Prompt Injection into Your SDLC

Six months ago, most teams' AI questions were about developers using chat assistants. Now a growing number are building products on top of large language models: support assistants, internal search over documentation, summarizers wired into ticket queues. The prototypes came together in an afternoon. The production questions are taking longer.

Two things changed this summer. On July 18, Meta released Llama 2, free for research and commercial use, with Microsoft as a preferred partner and availability through Azure. Running a capable model yourself is now a realistic option, which means more of the stack is yours to operate and secure. And in early August, OWASP published version 1.0 of its Top 10 for Large Language Model Applications.

The message from both is the same: an LLM application is an application. It has inputs, outputs, dependencies, permissions and failure modes. It belongs in your software development lifecycle like everything else.

Prompt injection is untrusted input in a new shape

The OWASP Top 10 for LLM Applications ranks prompt injection first. In a direct injection, a user crafts input that overrides the system prompt. In an indirect injection, hostile instructions arrive through content the model reads: a web page, an email, a document in your retrieval index.

If that sounds like SQL injection, it should, with one uncomfortable difference: there's no equivalent of a parameterized query. Instructions and data travel in the same channel, as natural language. You can reduce the risk. You can't reliably eliminate it with input filtering alone.

That shifts the design question from how to stop every malicious prompt to what the worst outcome is when one gets through. Answer that honestly and your architecture decisions get much clearer.

Threat-model the whole system, not just the model

Several items on the OWASP list are familiar risks in new clothes. Insecure output handling (LLM02) is what happens when model output is passed to a shell, a browser or a database without validation. Supply chain vulnerabilities (LLM05) cover the models, datasets and plugins you pull in. Sensitive information disclosure (LLM06) is your data leaking through responses. Excessive agency (LLM08) is giving a model more permissions or autonomy than its job needs.

A useful threat-modelling session for an LLM feature walks the data flow and asks:

• What untrusted content can reach the prompt, directly or through retrieval?
• Where does model output go next, and is it treated as untrusted input there?
• Which tools, APIs or credentials can the application invoke on the model's behalf, and are they scoped to the minimum?
• What sensitive data sits in the context window or the retrieval index, and who is allowed to see it?
• Which model, weights and libraries do we depend on, where did they come from, and who updates them?
• Which actions need a human to confirm them before they happen?

If you're self-hosting an open model, add the operational questions: where the weights are stored and how they're verified, how the inference endpoint is authenticated, how it gets patched, and how you'll handle abusive traffic that drives up GPU load. OWASP lists model denial of service as LLM04 for a reason.

Put evals in CI, next to your tests

Traditional tests check that a function returns the right value. LLM features are probabilistic, and their behaviour shifts when you change a prompt, a model version, a retrieval setting or a temperature. Without automated checks, every one of those changes is an untested deploy.

Treat evaluations as part of the build:

• A versioned eval set. Representative inputs with the expected properties of a good answer, stored in the repo alongside the prompt templates.
• Adversarial cases. Known injection attempts, both typed directly and embedded in documents, with assertions that the application refuses or contains them.
• Output contract checks. If the model must return JSON, validate the schema. If it must never emit a URL or a SQL statement, test for that.
• Pinned model versions. Treat a model or prompt change like a dependency upgrade: in a pull request, with eval results attached.
• Thresholds that block merges. A drop in pass rate on the eval set should fail the pipeline, just as a failing unit test would.
• Production logging with redaction. Capture prompts and responses, minus sensitive fields, so you can find failures and turn them into new eval cases.

Adoption is outrunning process

Developers aren't waiting for governance. In the Stack Overflow 2023 Developer Survey, 70% of all respondents said they are using or planning to use AI tools in their development process this year. On July 20, GitHub opened the Copilot Chat beta to all Copilot for Business users in Visual Studio and VS Code. The tooling is arriving everywhere at once. The guardrails mostly aren't yet.

What we recommend

Don't invent a separate, special process for AI features. Extend the one you have. Add LLM-specific questions to your threat-modelling template, using the OWASP list as the checklist. Treat model output as untrusted input everywhere it flows. Give the application the narrowest permissions that let it do its job, and require human confirmation for anything destructive or irreversible. Put evals in CI so prompt and model changes are reviewed like code.

If you're self-hosting Llama 2 or another open model, plan for it the way you would any stateful production service: authentication, patching, capacity, monitoring and a named owner.

If your team is moving an LLM feature from prototype to production and wants help with the pipeline around it, talk to us.