Threat modeling for AI means drawing your system, listing how someone could attack or misuse it, and deciding what you will do about each path before it ships. The method is the same as for any software. What changes is

the list of threats (prompt injection, poisoned data, over-powered agents) and the fact that you can’t assume any single defense works every time. A threat model tells you where to look. A code and infrastructure review shows whether what you built matches it. We do the second part at Bowtie, for founders and teams who need the answer in plain language.

TL;DR

  • Threat modeling is a structured way to find security risks in a system early, and AI adds new risks: prompt injection, data poisoning, model extraction and agents that act with too much power.
  • Map four layers first: application, data, model and infrastructure. Then mark every trust boundary, meaning every point where untrusted input crosses into something that can act.
  • Because AI output is probabilistic, plan to limit the blast radius of a failure rather than hoping to block every bad input.
  • Rank threats by likelihood and impact, assign an owner and a ticket to each, and re-run the model whenever your tools, data or permissions change.
  • A model on paper isn’t proof. Check it against the real code, the pipeline and the deployed infrastructure.

Table of Contents

What threat modeling means for an AI system

Threat modeling is a structured way to identify, understand and address security risks in an application and to communicate them to the people who decide. AWS describes it as a fundamental part of the design phase, because it lets you make security decisions early, when changes are cheap.

For AI, the OWASP AI Testing Guide says the same process reveals emerging threat vectors, clarifies attack paths against your data assets, and quantifies technical and business impact. It also stretches beyond code. It asks about regulatory and ethical context, and about how an attacker might manipulate inputs, poison training data or exploit bias.

Why does ordinary threat modeling fall short? Classic models look for flaws in deterministic software: the same input gives the same output, so a fix stays fixed. Microsoft’s security guidance frames AI threat modeling as an ongoing engineering discipline for probabilistic, agentic systems, with a focus on limiting the blast radius when something fails, because you can’t assume complete mitigation for every input.

If you are a founder or a CTO, think of it as three questions you can answer in a meeting:

  1. What are we building, and what can it touch?
  2. How could someone abuse it?
  3. What stops them, and how do we know?

Who needs this, and when

You need a threat model if your product does any of the following:

  • Sends user text to a language model and shows or acts on the answer.
  • Lets a model call tools: databases, email, payment systems, a CRM, a code repository.
  • Retrieves your own documents to answer questions (retrieval-augmented generation, or RAG).
  • Was built largely with AI coding tools and has never had a security review.
  • Runs a model on your own servers for privacy or compliance reasons.

The best time is before the build. The CD Foundation’s CI/CD security guide recommends AI-specific threat modeling during the code and prebuild phase, before components are implemented or wired into the pipeline. The second best time is now, on what you already have. A prototype that grew into a product without a design review is the most common place we see skipped security work, and it’s fixable.

Map the system before you list threats

A threat model is only as good as the diagram under it. The OWASP guide recommends splitting an AI system into four layers: application, data, model and infrastructure. Then trace how data moves between them and where trust changes.

For a typical AI feature, write down:

Layer What to list Question to ask
Application User interfaces, APIs, system prompts, session context Who can send input, and what can that input influence?
Data Training data, fine-tuning sets, retrieval documents, logs Who can add or change this, and is it ever treated as instructions?
Model Base model, fine-tuned versions, inference endpoint Can someone extract, probe or swap it?
Tools and agents Tool permissions, credentials, agent identities, connected systems If the model is tricked, what is the worst thing it can do?
Infrastructure Hosting, secrets, CI/CD pipeline, monitoring Who can reach it, and would we notice abuse?

An AI practitioner’s walkthrough on Medium adds system prompts, tool-grant policies and session context to the asset list. That is a useful reminder: the instructions you gave the model are an asset, and so are the permissions you handed it.

Then draw the trust boundaries. A trust boundary is any line where data passes from a less trusted place to a more trusted one: a user’s message entering your prompt, a web page your agent reads, a document uploaded into your retrieval index. The STRIDE-AI research paper goes as far as tagging these as probabilistic trust boundaries, because the model’s behavior at them can’t be fully predicted.

The SANS Institute’s AI Defense Matrix, released in 2026, sorts AI into eight asset classes. Two of them are worth noting for small teams: AI-generated code and AI agent identities. Both are easy to forget when you list assets.

The AI-specific threats to cover

You don’t need to memorize every taxonomy. You need a checklist that fits your system. Finland’s national cybersecurity authority, Traficom, lists the threats a model should cover: prompt injection, data and memory poisoning, model inversion, model extraction, and agent risks such as cascading failures, tool abuse, identity and privilege abuse, and collusion between agents. It also notes that agents and the language models under them can’t reliably tell instructions apart from content an attacker supplies. That one fact explains most of the list.

Here is what each means in plain terms:

  • Prompt injection. Someone hides instructions in a message, a web page or a document, and the model follows them. It is the first entry in the OWASP Top 10 for LLM Applications (2025 edition).
  • Sensitive information disclosure. The model reveals customer data, secrets or your system prompt.
  • Data and model poisoning. Bad data gets into training, fine-tuning or your retrieval index, and quietly changes behavior.
  • Excessive agency. An agent has more permissions than its job needs, so one successful trick becomes a real incident.
  • Supply chain. A model, library or plugin you pulled in is compromised.
  • Improper output handling. Your app trusts what the model returns and passes it straight to a database, a shell or a browser.
  • Unbounded consumption. An attacker, or a bug, runs up your token bill or takes your service down.

For agents, AWS’s prescriptive guidance points out that many threats target the reasoning layer, through input flooding, logic manipulation and multi-agent collusion, and that probabilistic decision-making calls for specialized mitigations.

A five-step process you can run in a week

A workable process for a small team follows the same shape as the published methods. The steps below combine the structure in OWASP’s DevSecOps guideline, which rests on four questions (what are we working on, what can go wrong, what will we do about it, did we do a good enough job), with the AI-specific stages above.

Step 1: Define scope and assets

Pick one feature or workflow, not the whole company. List the model, any fine-tuned versions, the retrieval corpus, the inference API, every tool the agent can call, and the libraries underneath. Right-size it: OWASP’s guidance is to focus on the risky features instead of modeling everything.

Where it goes wrong: teams skip the tools and credentials, which is where the damage happens. If you built with AI coding assistants, include the generated code as an asset. Nobody may have read it line by line.

Step 2: Draw data flows and trust boundaries

Sketch how a request travels from the user to the model, to your tools, and back. Mark each trust boundary. Keep the diagram in version control next to the code so it changes when the system changes.

AI request flow with trust boundaries

Where it goes wrong: the diagram describes the design, not what was actually deployed. Compare it with the real configuration.

Step 3: Enumerate threats at each boundary

Walk each boundary and ask what an attacker could supply, what they could read and what they could trigger. STRIDE is the familiar frame (spoofing, tampering, repudiation, information disclosure, denial of service, elevation of privilege). The STRIDE-AI paper adapts it: tampering includes contaminating training data, and elevation of privilege can show up as jailbreaking. Pair it with the AI threat list above.

Where it goes wrong: brainstorming only the attacks you’ve read about. Include honest misuse too, such as an employee pasting confidential data into a prompt.

Step 4: Score and decide

Rate each threat on likelihood and impact, write down your assumptions, then pick a response: remove it, reduce it, accept it or transfer it. For a financial view, the FAIR-AIR approach quantifies AI risk in dollar terms, using how often a loss event happens and how large it is.

The mitigations that come up again and again in practitioner guidance are architectural, not clever prompts:

  • Keep system instructions separate from untrusted content.
  • Require human approval for high-risk actions, such as refunds, deletions or external emails.
  • Sandbox agents and give each one the narrowest permissions that work.
  • Filter and validate model output before another system acts on it.

Every threat gets a ticket and an owner. OWASP recommends a backlog item for each identified threat, so findings don’t die in a slide deck.

Step 5: Validate and repeat

Test whether the threats you identified are actually exploitable given your controls. AWS says threat models should inform automated pipeline tests and help define the scope of penetration testing. Then update the model whenever a change crosses a trust boundary: a new tool, a new data source, a new model version.

Prioritizing when everything looks risky

A first model often produces forty findings. Rank them with three filters:

  1. Reach. Can an anonymous user trigger it, or only an authenticated admin?
  2. Blast radius. If it works, what can the attacker read, change or spend?
  3. Reversibility. Can you undo the damage? A leaked prompt is bad. A deleted production table or a sent wire transfer is worse.

A worked example: a support bot reads help-center articles and can issue refunds through an internal API. An attacker plants instructions in a public forum post that the bot retrieves. Reach is high, since anyone can post. Blast radius is financial, and reversibility is poor once money moves. That one path outranks ten harmless prompt-leak findings. The fix isn’t a smarter prompt. Take refunds out of the bot’s direct control, or require human approval above a threshold, and treat retrieved text as untrusted.

For a broader frame, the NIST AI Risk Management Framework organizes the work into four functions: Govern, Map, Measure and Manage. NIST says AI systems should be tested before deployment and regularly while in operation. Its Generative AI Profile, released in July 2024, applies that approach to generative systems. Both are voluntary in the U.S., but they give your board, insurer or enterprise customers a shared vocabulary. A documented threat model is also the kind of evidence a due-diligence questionnaire asks for.

How to tell whether your threat model works

A model that nobody tests is a document. Four checks separate a useful one from shelfware:

  • Coverage. Does every trust boundary and every tool have at least one threat against it?
  • Regression sensitivity. If you add a tool or change a prompt, does a test fail or a review trigger?
  • Operational fit. Do engineers use it in design reviews, or did it end up in a folder?
  • Change resilience. When did you last update it, and what triggered the update?

These track the testing and evaluation criteria (coverage, sensitivity, operational fit) that compliance guidance mapped to the NIST framework suggests for judging whether your testing is effective. Ownership matters too. AWS notes that threat modeling is a human activity across business, builder and security teams, and that the people closest to building and running the application should own the model.

We cover the engineering side of this in our guides on QA for AI applications, AI observability and AI agent security.

From model to proof: what to check in the code and infrastructure

The threat model produces a list of questions. The answers live in the repository, the pipeline and the servers. For a team without a dedicated security engineer, this is the gap that stays open.

Take the common cases:

  • You built the app with AI coding tools. The model says secrets must not reach the client, and the real code is in front of you. Does a hard-coded key sit in a config file? Are database queries parameterized? Is user input passed to the model and then to a tool with no checks?
  • Your agent has tool access. The model says permissions should be minimal. Does the service account actually hold admin rights because that was the quickest way to get the demo working?
  • Your pipeline ships changes. The model says changes crossing trust boundaries need review. Does your CI run anything that would catch one?
  • You’d notice an attack. The model says abuse should be detectable. Are logs, alerts and token-spend limits in place?

Our production hardening guide for AI apps and our write-up on code audits and isolation for agentic AI governance go deeper on these checks. If your regulator is a bank or broker, mapping NIST to OCC and FINRA expectations is the next read.

Where Bowtie fits

We review and repair AI-built software for a living, and we publish prices so you can budget before you call. For the verification half of threat modeling, these are the entry points:

  • ShipDoctor scan, $59 one-off. A fast first look at code quality, performance and security.
  • Vibe Check, $449 one-off. A review of an AI-generated or vibe-coded app, made for nontechnical founders who need to know what they actually have.
  • Senior Developer Review, from $449 one-off. An experienced engineer reads your code.
  • Infrastructure Assessment, $2,750 one-off, and Continuous Integration (CI) Assessment, $4,500 one-off. They look at the servers and the pipeline your threat model assumes are sound.
  • Automated Monitoring / Logging / Alerts, $2,750 one-off. So that abuse of the system shows up somewhere a person will see it.

If you want ongoing help, our Ship Shape Service starts at $1,850 per month and the Crew Service at $2,950 per month. We also build AI agents and workflow automation and deploy local AI models for teams whose data has to stay in-house.

Bowtie

Start with the Vibe Check and monthly plans if you have an app built with AI tools, or see the full price list and our services for the rest.

Our perspective

Most of the AI projects that reach us didn’t fail because the team ignored security. They stalled at the last mile: a prototype worked, someone said “ship it,” and nobody went back to ask what the agent could reach, who could talk to it and what happened when it was wrong.

Our view is simple. A threat model is the cheapest security work you will ever do, and it pays off only if it connects to real code. Draw the system, mark where untrusted text meets something powerful, and shrink what that something can do. Then have an experienced engineer read the code that actually runs. Clean, secure, production-ready code is not a slogan for us. It’s what the review is for.

We pair AI tooling with senior human engineers because the combination catches what either misses. And we stay after launch, because a model you wrote in March is out of date by the time you add a new tool in June.

Chad

FAQ

Is threat modeling for AI different from regular threat modeling?

The process is the same: scope, diagram, find threats, decide, validate. The content differs. You add AI threats such as prompt injection, data poisoning and model extraction, and you accept that model behavior is probabilistic. That is why guidance from Microsoft and AWS stresses limiting the blast radius of a failure.

Which framework should a small team start with?

Start with the four questions in OWASP’s threat modeling guidance and the OWASP LLM Top 10 as your threat checklist. Add STRIDE if your team already knows it. Use the NIST AI RMF when customers or regulators ask how you govern AI risk.

How often should we update an AI threat model?

Whenever something crosses a trust boundary: a new tool or integration, a new data source, a new model or major prompt change, or a new group of users. OWASP’s guidance is to update continuously and keep the model in version control. A scheduled review each quarter is a reasonable backstop.

Do we need a threat model if our app is small or vibe-coded?

Yes, and a small scope makes it quick. Model one workflow, list what the AI can touch, and cut its permissions. If the app was generated by AI tools, a code review shows what the app really does, which is often different from what you intended.

Sources