How To Conduct Threat Modeling For Large Language Models

Large language model incidents usually happen at the seams, not inside the model weights. A user prompt turns into a document lookup, a tool call, a database query, or a logged transcript—and that is where sensitive data leaks and unsafe actions start.

Primary focus	Threat modeling for LLMs as a system-level security process as of October 2026
Best used for	Chatbots, copilots, RAG systems, and agentic workflows as of October 2026
Main risks	Prompt injection, data leakage, retrieval poisoning, tool abuse, and identity failures as of October 2026
Method	Map flows, define boundaries, test abuse cases, rank risks, and apply controls as of October 2026
Related framework	OWASP Top 10 for Large Language Models as of October 2026
Reference standard	NIST Artificial Intelligence Risk Management Framework as of October 2026
Supporting course	OWASP Top 10 For Large Language Models (LLMs) from ITU Online IT Training as of October 2026

Introduction

Threat modeling for LLMs is the process of identifying where a large language model system can be attacked, misused, or tricked across the full stack, not just inside the model itself. That includes the user interface, orchestration layer, retrieval pipeline, tools, logs, identity systems, and downstream automations.

The reason this matters is simple: many LLM failures happen when a harmless-looking prompt crosses a trust boundary. A chatbot that summarizes a file can accidentally expose confidential content if retrieval is mis-scoped. An agent that can send email or open tickets can turn a bad instruction into a real-world action.

This framework is for AI engineers, security teams, product leaders, privacy reviewers, and compliance stakeholders who need a repeatable way to assess risk and prevent AI data breaches. It is especially useful for copilots, chatbots, retrieval-augmented generation systems, and agents that connect to internal systems.

Threat modeling an LLM without mapping retrieval, tools, logging, and identity is like reviewing a firewall rule while ignoring the rest of the network.

For a broader governance lens, NIST’s Artificial Intelligence Risk Management Framework is a useful reference point, and the OWASP Top 10 for Large Language Models gives security teams a practical taxonomy for common failure modes.

Why Is Threat Modeling For LLMs Different From Traditional Application Threat Modeling?

LLM systems behave differently from classic applications because they are probabilistic, instruction-sensitive, and often connected to external data and tools. Traditional software usually follows deterministic code paths. An LLM may produce a different answer from the same input, which makes risk analysis more about behavior patterns than fixed outputs.

That difference changes how security teams think about abuse. In a traditional app, a user may try to break authentication or bypass access control. In an LLM app, the user may also try to override system instructions, inject malicious content into retrieved documents, or manipulate the model into calling a tool it should not use.

The attack surface also expands. A prompt can become a retrieval query, then a context bundle, then a tool invocation, then a ticket, email, or database update. Security review has to follow the chain all the way through the data pipeline and into the orchestration layer.

• Classic app risk: A user tries to access a page they should not see.
• LLM-specific risk: A user or document tries to change the model’s instructions.
• Classic app risk: A form field submits invalid input.
• LLM-specific risk: A prompt hides instructions that target the model’s behavior.

CompTIA’s security guidance and the OWASP LLM material both reinforce the same lesson: application controls still matter, but they are no longer enough by themselves.

Understanding The Full LLM System And Its Attack Surface

An attack surface is every place an attacker can influence, observe, or misuse a system. For an LLM application, that surface includes the user interface, system prompt, middleware, model API, retrieval index, connectors, plugins, logs, and any backend action the model can trigger.

It helps to treat each component as a separate trust boundary. A user may be trusted to ask questions but not to see private files. A retrieval index may be trusted to search approved content but not to expose raw source documents. A tool may be trusted to read a CRM record but not to send an outbound email without confirmation.

Adding business systems increases the blast radius quickly. Once the LLM can touch email, file shares, ticketing systems, cloud APIs, or internal databases, a prompt is no longer just text. It becomes a request that can move data or trigger a workflow.

Typical LLM components to map

• User interface: Chat window, API endpoint, or embedded assistant.
• Orchestration layer: Code that builds prompts, selects tools, and handles responses.
• Model provider: The hosted or self-managed model endpoint.
• Retrieval system: Search, ranking, and context assembly for RAG.
• Tools and connectors: Email, file storage, databases, ticketing, SaaS APIs.
• Logging pipeline: Prompt logs, transcript storage, analytics, and monitoring.

The practical rule is straightforward: if a component can read sensitive content or act on behalf of a user, it belongs in the threat model. That is why the most important security issues often happen outside the model itself.

Prerequisites

Before you start, make sure the team has enough context to model the system honestly. Missing information leads to weak controls and false confidence.

• System diagram: A current architecture view of the LLM application, including retrieval, tools, and logging.
• Access to engineers: Someone who knows how prompts, connectors, and backend actions actually work.
• Data inventory: A list of content sources, secrets, user data, and regulated data types.
• Security and privacy stakeholders: People who can judge risk, retention, and compliance requirements.
• Test environment: A safe place to validate prompts, retrieval behavior, and tool permissions.
• Reference framework: OWASP Top 10 for Large Language Models and NIST AI RMF as baseline guidance.

NIST AI RMF is useful here because it pushes teams toward structured governance, not just one-off testing. That matters when the same LLM design will be reused across products or business units.

How Do You Map Data Flows Before You Assess Risk?

Data-flow mapping is the process of tracing how prompts, context, outputs, and telemetry move through the LLM system. You cannot rank risk accurately until you know where data enters, where it is transformed, where it is stored, and who can access it.

Start with the user prompt. Then trace how the application wraps that prompt with a system message, fetches retrieved context, sends it to the model, receives a response, and stores the result. Do the same for transcripts, analytics events, feedback data, and any background jobs that reuse conversation history.

This step often exposes hidden copies of sensitive data. A prompt may be deleted from the chat UI but still persist in logs, observability platforms, exports, or support tickets. The same is true for retrieved content that gets cached or indexed elsewhere.

1. Record the source: Identify every origin of content, including user input, system prompts, files, and databases.
2. Record the destination: Document every place that content goes, including the model, logs, caches, and third-party services.
3. Assign a trust level: Mark whether the source is user-controlled, internal, vendor-managed, or external.
4. Set a retention window: Note how long the data stays in each store and who can delete it.
5. Mark sensitive fields: Flag secrets, personal data, regulated content, and proprietary prompts.

This is the point where many AI data breach prevention efforts either become real or fail quietly. If the team cannot describe the flow, the team cannot control the flow.

What Assets, Trust Boundaries, And Security Objectives Should You Identify?

Assets are the things you need to protect. In an LLM system, that includes proprietary prompts, private documents, user data, internal workflows, API keys, embeddings, model outputs, and the business logic encoded in orchestration code.

Trust boundaries are just as important. A boundary exists whenever data moves between users, application services, retrieval systems, model providers, or external tools. Once data crosses a boundary, the assumptions about access, handling, and reliability change.

The three classic security objectives still apply, but they need LLM-specific context. Confidentiality means the model should not reveal private content. Integrity means the system should not be tricked into making bad decisions or changing approved instructions. Availability means the chatbot, agent, or retrieval service should keep working under load or attack.

Asset	Why it matters
System prompts	Can reveal policy, workflow, or hidden business logic if exposed.
Private documents	May contain confidential, regulated, or restricted internal information.
API keys and tokens	Can allow unauthorized access to downstream systems.
Conversation memory	Can persist sensitive content across sessions or users.

Asset criticality changes by use case. A support chatbot can be damaging if it leaks customer data. A coding assistant can be damaging if it exposes source code or secrets. A legal-review assistant can be damaging if it surfaces privileged material to the wrong user.

How Do You Build A Threat Model Around Realistic Abuse Scenarios?

Abuse scenarios are attacker stories, not abstract weaknesses. They make the threat model concrete because they show how a real person or malicious document could exploit the system end to end.

Use scenarios such as prompt injection, indirect prompt injection from retrieved content, unauthorized tool use, and data extraction from logs or conversation memory. Then ask what the attacker wants, what the system lets them touch, and what the worst outcome would be.

For example, an attacker might upload a document that tells the model to ignore prior instructions and reveal the contents of hidden context. Another attacker might try to convince the model to send an email, create a ticket, or query a database with parameters that expose data they should not see.

The best LLM threat models are written as stories about abuse, not as lists of generic risks with no operational context.

• Scenario: A malicious document in the retrieval corpus injects hidden instructions.
• Scenario: A user tries to coerce the assistant into revealing the system prompt.
• Scenario: The model is tricked into using a tool with excessive permissions.
• Scenario: An internal report is generated with data the requester should not access.

The OWASP Top 10 for Large Language Models is useful here because it helps teams standardize the language around these abuse cases instead of treating them as one-off bugs.

How Do Prompt Injection, Indirect Injection, And Instruction Conflicts Work?

Prompt injection is an attack that tries to override or redirect model behavior by inserting malicious instructions into user input or retrieved content. The direct version comes from the user. The indirect version comes from documents, web pages, emails, or tool outputs that the model reads as context.

Instruction conflicts happen when untrusted content competes with system rules or business logic. If the orchestration layer does not clearly separate trusted instructions from untrusted text, the model can follow the wrong instruction chain.

Warning signs are often visible in testing. The model may ignore policy language, unexpectedly call a tool, summarize confidential content it should not have seen, or echo hidden instructions from a retrieved document. Those symptoms should be treated as security failures, not just weird model behavior.

1. Test direct injection: Try prompts that ask the model to ignore previous instructions or reveal hidden context.
2. Test indirect injection: Place malicious instructions inside retrieved documents or sample web content.
3. Test conflict handling: Check whether the model obeys system rules when context contains contradictory instructions.
4. Test tool boundaries: Verify that the model does not escalate from text manipulation to real action.

Microsoft’s official guidance for building secure AI applications is helpful here because it emphasizes prompt handling, content filtering, and application-layer controls rather than relying on model behavior alone.

How Do You Assess Data Leakage Risks Across The Entire Stack?

Data leakage is the unauthorized exposure of sensitive content through prompts, responses, logs, embeddings, cached outputs, or retrieved documents. In LLM systems, leakage often happens because the model sees more than the end user should see.

Look at every location where private data can persist. That includes raw prompts, conversation transcripts, analytics events, vector databases, backup systems, and support tooling. A well-designed assistant can still leak data if telemetry is too verbose or retention is too broad.

Retrieval systems deserve special attention. If access control is weak, the model may fetch internal documents that belong to a different role, team, or customer. The same problem can appear when embeddings are built from content that should never have been indexed in the first place.

For privacy-heavy environments, the right question is not “Can the model answer?” It is “Should this content enter the context window at all?” That mindset is central to AI data breach prevention.

How Do You Evaluate Tool Use, Agents, And Downstream Automation?

Tools are execution surfaces. Once an LLM can read a file, send a message, create a ticket, or query a database, model mistakes can create real-world impact. That is why tool permissions must be reviewed with the same care as any privileged service account.

Start with least privilege. Give each tool only the permissions needed for the business task. If an action is sensitive, require human confirmation before execution. That is especially important for agents that can plan multiple steps in a row.

Agentic workflows increase risk because one wrong step can cascade. A bad retrieval result can influence a plan, the plan can select the wrong tool, and the wrong tool can trigger a downstream action that looks legitimate in logs.

• Read access: Can the tool see only approved records or entire datasets?
• Write access: Can it change data, or only propose changes for review?
• Network access: Can it call external endpoints or only internal services?
• Approval controls: Does a human need to confirm high-impact actions?

If the model is tricked into choosing the wrong tool, the failure is not just a bad answer. It is a workflow breach. That distinction matters when the system can create tickets, send customer messages, or update production records.

How Do Identity, Authentication, And Authorization Affect LLM Security?

Authentication is how you verify the user, and access control is how you decide what that user can do. In LLM systems, both must hold across the front end, retrieval layer, tools, and backend systems.

Identity should not disappear once a request leaves the chat interface. If the application cannot preserve who the user is all the way through retrieval and tool execution, authorization checks become inconsistent. Shared accounts and weak session handling make this much worse.

Authorization needs to happen more than once. The application layer should validate what the user is allowed to ask for. The data source layer should validate what records or files can actually be read. If either layer is missing, the assistant can become an unintended proxy for broader access.

1. Verify the user: Confirm the application uses strong sign-in and session controls.
2. Preserve identity: Carry the user context into retrieval, tools, and audit logs.
3. Enforce authorization twice: Check permissions in the app and at the data source.
4. Review service credentials: Remove overbroad API keys, shared tokens, and stale secrets.

When an LLM can act on behalf of a person or team, identity is not a side issue. It is the control plane for the entire security model.

How Do You Test Retrieval Systems For Poisoning And Unauthorized Exposure?

Retrieval-augmented generation is powerful because it lets the model use external content, but it also introduces a new class of risk. A poisoned document can influence the model’s behavior, and an overbroad retrieval rule can expose content to the wrong user.

Test the ingestion pipeline first. Check where documents come from, who can upload them, how they are validated, and whether they are tagged with trust levels. If any source can inject content into the retrieval corpus without review, the model can be manipulated through the back door.

Then test access behavior. A user should not receive content they would not otherwise be allowed to open. Retrieval quality and retrieval security are related, but they are not the same thing. Good search results do not excuse bad authorization.

• Source trust: Is the document from a trusted repository or a user-uploaded source?
• Validation: Are malicious instructions stripped or flagged during ingestion?
• Access filtering: Is content filtered by user permissions before it reaches the model?
• Provenance: Can the system explain where the retrieved content came from?

If retrieval can surface sensitive documents without proper access control, the model becomes a disclosure engine. That is a serious confidentiality failure, even when the answer sounds helpful.

How Do Logging, Monitoring, And Incident Response Fit In?

Logging is useful only if it helps you investigate issues without creating a bigger privacy problem. In LLM systems, logs often contain prompts, completions, usernames, document snippets, and tool outputs, so they must be designed carefully.

Review what is logged, where it is stored, how long it is retained, and who can access it. If logs include secrets or regulated data, they become a secondary sensitive system and must be protected accordingly. The same is true for metrics and analytics pipelines that copy message content for reporting.

Monitoring should detect abnormal behavior, not just uptime. Repeated policy violations, unusual tool use, sudden spikes in retrieval volume, or attempts to extract hidden context are all worth alerting on. Incident response should define what happens when a prompt leak, tool abuse, or integration compromise is discovered.

IBM’s data breach research and the Verizon Data Breach Investigations Report both show a consistent pattern: weak visibility and delayed response make incidents more expensive and harder to contain.

How Do You Prioritize Risks And Rank Mitigations?

Risk ranking is the step that turns a threat model into an execution plan. Without prioritization, teams collect findings but do not reduce exposure in any meaningful way.

A simple method works well: score each threat by likelihood, impact, and exploitability. A low-probability issue with catastrophic consequences may deserve higher priority than a common but minor annoyance. The point is to focus on what can actually hurt the business.

Start with threats that can cause data loss, unauthorized actions, or cross-system compromise. Then assign a clear owner to each mitigation. If nobody owns the fix, the threat model becomes shelfware.

Threat	Typical priority signal
Prompt injection	High if the system accepts external or untrusted content.
Data leakage	High if the system handles private or regulated data.
Tool abuse	High if the model can write, send, or trigger actions.
Retrieval poisoning	High if many sources can feed the retrieval corpus.

Keep the model current. A new prompt template, a new connector, or a new business case can change the risk profile overnight.

What Controls Reduce LLM Security Risk?

Controls should reduce both the chance of failure and the impact if something goes wrong. The strongest programs use layered defenses rather than one big control that is expected to solve everything.

Start with input validation, output filtering, and prompt hardening. Limit tool permissions, and require explicit approval for sensitive actions. Use data minimization so the model receives only the context it needs for the task at hand.

Then segment the environment. High-value data, secrets, and internal workflows should not be broadly reachable from the assistant. Isolation matters because one compromised integration can become a path into many systems.

1. Validate inputs: Reject malformed, malicious, or unsupported prompt content.
2. Harden prompts: Separate trusted instructions from user-controlled text.
3. Restrict tools: Limit actions, scope, and permissions to the minimum needed.
4. Minimize context: Pass only the data the model truly needs.
5. Monitor abuse: Add policy checks, red-teaming, and alerting before release.

The OWASP Top 10 for Large Language Models course content from ITU Online IT Training aligns well with this control set because it focuses on practical defensive patterns rather than theory alone.

How Do You Create A Repeatable Threat Modeling Workflow For Teams?

Repeatability is what turns threat modeling from a workshop into a security process. If the team cannot run the same method again next month, it will not scale across products or releases.

Begin with scoping. Define exactly which application, users, models, data sources, and tools are in scope. Then run a data-flow workshop with engineering, security, product, privacy, and compliance stakeholders so everyone sees the same system.

From there, brainstorm abuse cases, assign mitigations, and track them to closure. The model should be updated whenever prompts, retrieval sources, tools, or business requirements change. That is normal, not a failure.

1. Scope the system: Decide what is inside and outside the model.
2. Map the flows: Document prompts, retrieval, tools, logs, and storage.
3. Identify the threats: Use realistic abuse cases instead of generic guesses.
4. Assign controls: Tie each mitigation to an owner and deadline.
5. Review changes: Reassess the model after every major release or integration change.

A repeatable workflow is the difference between a one-time review and a mature AI security practice.

What Are The Most Common Mistakes To Avoid?

The most common mistake is treating the model as the only thing that can fail. In reality, the surrounding application, logging, retrieval, and identity layers are often the weakest links.

Another mistake is assuming prompt injection is the only meaningful threat. It is important, but it is not the only problem. Retrieval poisoning, over-permissioned tools, stale logs, and weak authorization can be just as damaging.

Teams also make the mistake of trusting vendor-managed models without reviewing their own design. Even if the model provider is strong, the local application can still leak data or trigger unsafe actions. Finally, a threat model should never be a static document that gets filed away after launch.

• Do not: Focus only on the model and ignore the app layer.
• Do not: Treat prompt injection as the whole problem.
• Do not: Skip retrieval security and authorization review.
• Do not: Assume vendor hosting removes internal risk.
• Do not: Leave the model stale after release.

The secure approach is maintenance, not ceremony. LLM threat modeling must stay alive as the system evolves.

How Does OWASP Top 10 For Large Language Models Help Structure The Process?

OWASP Top 10 for Large Language Models gives teams a common language for the most important categories of LLM risk. That helps engineers, security staff, and reviewers discuss issues without talking past one another.

The value of a framework is consistency. If every team uses the same categories to evaluate prompts, data, tools, and outputs, reviews become easier to compare and easier to audit. That is especially useful when multiple copilots or agents are being built across the same organization.

OWASP-style categories also support better remediation. Instead of saying “the chatbot is risky,” the team can say “this system is vulnerable to indirect prompt injection through retrieved documents” or “this agent has excessive tool permissions.” That level of specificity leads to actionable fixes.

OWASP Top 10 for Large Language Model Applications is worth using as a checklist, but it works best when combined with a full system threat model. Categories are useful. Context is what makes them practical.

How Do You Verify It Worked?

Verification means proving the threat model changed something real. A completed workshop is not enough unless the system is safer, the owners are assigned, and the team can demonstrate control behavior.

Look for concrete signs: the architecture diagram is updated, data flows are documented, sensitive sources are tagged, tool permissions are reduced, and the highest risks have owners and deadlines. In testing, the assistant should resist obvious prompt injection attempts, avoid exposing unauthorized data, and require confirmation for sensitive actions.

1. Check the artifact: Confirm the threat model covers users, data flows, tools, logs, and trust boundaries.
2. Check the fixes: Verify that mitigations were assigned and implemented where possible.
3. Check the behavior: Run abuse tests for injection, leakage, and unauthorized actions.
4. Check the logs: Confirm sensitive data is minimized and retention is documented.
5. Check the process: Ensure updates happen when prompts, tools, or data sources change.

Common failure symptoms include unsafe tool calls, unexpected retrieval results, persistent sensitive data in logs, and answers that ignore instruction hierarchy. If any of those appear, the model is not done.

Conclusion

Effective threat modeling for LLMs requires you to view the whole system as the attack surface. The model is only one piece. The rest of the risk lives in prompts, retrieval, tools, logs, identity, and downstream automation.

The biggest threats are predictable: prompt injection, data leakage, retrieval poisoning, tool abuse, and identity failures. Those risks become much easier to manage when the team maps data flows, defines trust boundaries, tests realistic abuse cases, and ranks mitigations by impact.

Make threat modeling part of design, launch, and monitoring. That is how you move from reactive cleanup to informed security decisions. It is also how you prevent an AI mistake from becoming an AI incident.

If your team is building chatbots, copilots, agents, or RAG systems, use this framework alongside the OWASP Top 10 for Large Language Models course from ITU Online IT Training to turn security review into a repeatable practice.

CompTIA®, Microsoft®, and OWASP are referenced for educational and descriptive purposes. Trademarks belong to their respective owners.