An AI runbook is a documented procedure written so an AI agent can read it, gather evidence, and carry out its steps, with a human approving anything that changes production. The hard part is not the writing. It is the
last mile: connecting the runbook to real systems, limiting what the agent can touch, and watching what it does. That is engineering work, and we build it with teams through our AI agent creation and workflow automation service.
TL;DR
- An AI runbook is a procedure for a known failure or task, written so an agent can follow it, not just a person.
- The safe pattern is read-only investigation first, then a proposed action, then human approval for anything that writes.
- Google’s SRE guidance asks for least-privilege agent identities, rate limits, circuit breakers, dry-run support and staged autonomy.
- Agents can fail without throwing an error, so you need observability on the agent itself, not only on the system it watches.
- Start with one well-understood, frequent incident, and expand only after the agent has earned trust.
Table of Contents
- What an AI runbook is, and how it differs from a classic one
- How agents use runbooks in practice
- Where teams use AI runbooks
- How to build an AI runbook from your existing procedures
- Guardrails: what safe looks like when a runbook can act
- Watching the agent: why observability comes with the runbook
- Common mistakes with AI runbooks
- Our perspective
- Getting help building yours
- FAQ
- Sources
What an AI runbook is, and how it differs from a classic one
A traditional runbook is a document: a checklist an on-call engineer opens at 3 a.m. when an alert fires. It says which dashboard to check, which command to run, who to page. It works when the person reading it has context and judgment, and it rots quietly when systems change.
An AI runbook keeps the same job but changes the reader. The reader is an agent, a language model with access to tools such as log search, a ticketing system, a code repository or a cloud console. That changes what the document must contain:
- Explicit triggers. Which alert or event starts the procedure.
- Ordered evidence gathering. What to look at first, second and third, so the agent does not free-associate.
- Decision criteria. What counts as a finding, and when to stop and escalate.
- Permitted actions. What the agent may do alone, what needs approval, and what it must never do.
- Output format. What the final report or ticket update should contain.
A classic runbook can lean on “check the usual suspects.” An agent cannot. It needs the usual suspects listed, with the tools to check them.
How agents use runbooks in practice
Two published examples from major engineering organizations show the pattern clearly.
Google Cloud’s architecture documentation for security operations describes a coordinator agent that fetches AI runbooks, skills and the incident response plan from a knowledge database, splits the request into subtasks, hands them to specialist sub-agents, and iterates until the findings meet the runbook’s evaluation criteria. Only then does it write up a report.
OpenAI’s SRE agent example works from an incident alert. The alert opens a Slack on-call thread, and the agent investigates production telemetry, recent GitHub pull requests and commits, AWS infrastructure and similar past incidents. It posts its findings and then pauses for a responder to approve any rollback. The surrounding application handles alert deduplication, responder permissions and incident history.
Both designs share three traits. The runbook defines the investigation. The agent gathers evidence before it proposes anything. A human or a defined evaluation gate decides what happens next.

Where teams use AI runbooks
Security operations teams were early adopters because their procedures are already written down. A Google Cloud community post from March 2026 describes splitting runbooks into modular agent skills: triage skills for alerts, suspicious logins and malware, response skills for ransomware, phishing and compromised accounts, plus enrichment and case-documentation skills.
The same idea applies well beyond security:
- Site reliability. Restarting a stuck worker, checking a failed deploy, correlating an error spike with a recent commit.
- E-commerce operations. Investigating a checkout failure or a theme change that broke a storefront. Our Shopify site alerts page covers the alerting side.
- Business workflows. Onboarding a customer, reconciling an order, routing a support ticket, where the “runbook” is a process document and the tools are your CRM and billing system.
If your goal is to take repetitive operational work off a small team, that last category is often the best starting point because the blast radius is smaller. Our guide to AI workflow automation covers it in depth.
How to build an AI runbook from your existing procedures
You do not need to start from a blank page. You need a procedure someone already trusts. Here is the routine, with the places it usually goes wrong.
Step 1: Pick one frequent, well-understood incident
Choose a failure that happens often, has a known cause list and a known fix. A disk filling up, a queue backing up, a certificate nearing expiry. Avoid the vague, once-a-year disaster for your first attempt.
Where it goes wrong: teams pick the scariest incident because it feels most valuable. An agent has no history with you yet, and that is the wrong place to test it.
Step 2: Rewrite the procedure for a reader with no context
Take the existing document and remove every phrase that depends on tribal knowledge. “Check the usual dashboard” becomes a named dashboard, a query, and what a bad result looks like. Name each system, environment and owner.
Where it goes wrong: the document says what to do but not how to tell it worked. Add a pass condition to every step.
Step 3: Split steps into read, propose and write
Label each step by what it does to the world:
- Read: query logs, fetch metrics, look up recent deploys. The agent runs these on its own.
- Propose: draft a rollback, a config change or a restart. The agent writes it up but does not execute.
- Write: the change that touches production. A person approves it.
This mirrors the open-source pattern of turning Markdown runbooks into agent-executable steps, where read-only diagnostics run automatically and anything that changes state waits for approval of that exact step.
Step 4: Connect the tools with narrow permissions
Give the agent its own identity and only the credentials the read steps need. Do not reuse an engineer’s token. Expose the minimum set of tools, and keep raw write tools out of what the model can see. LegalZoom Engineering describes filtering write tools out of the model’s toolset so the agent can only propose changes, which makes unsafe action structurally impossible rather than merely discouraged.
Where it goes wrong: the quickest integration is a broad API key. It works in the demo and becomes the incident later.
Step 5: Add context beyond the repository or the ticket
The same LegalZoom team found that agents fail when they only see the repo they are in. A fix can be correct locally and wrong for the architecture. Give the agent a map of how services relate, and guidance files at the repository level that state your conventions.
Step 6: Replay past incidents before going live
Run the runbook against old incidents with the write tools disabled. Compare the agent’s findings with what the responders found. Where they differ, fix the runbook, not the prompt.
Step 7: Release in stages
Start with the agent investigating and posting findings only. Move to proposing actions. Only later let it execute low-risk steps. Keep a log of every tool call and every approval.

Guardrails: what safe looks like when a runbook can act
Google’s SRE guidance on AI engineering for reliable operations is the most concrete public checklist we have seen. It calls for distinct least-privilege agent identities with no ambient access, agent-specific rate limits and circuit breakers, mandatory dry-run support on agent-facing APIs, and actuation through delegated control planes rather than direct access. It also asks for real-time risk evaluation of each proposed action against production context such as ongoing deployments, error budget, active incidents and time of day, and for progressive authorization, where agents start at human-approved autonomy levels and move up as they prove reliable.
Google Cloud’s security documentation adds human-in-the-loop review for business-critical systems, an agent gateway with default-deny traffic policies, and a registry so agents connect only to approved sub-agents and tools.
LegalZoom’s approach to approval is a good model for a small team. Approval is an explicit contract: scoped to the thread, limited to an authorized group, with a two-human rule and a stale-data check that re-reads current state before an approved change is applied. A yes given ten minutes ago about a system that has since changed should not still count.
Here is how those controls map to a practical checklist:
| Control | What it prevents | Where to start |
|---|---|---|
| Dedicated agent identity | Agent inheriting a human’s broad access | One service account per agent, read-only |
| Read, propose, write split | Unreviewed production changes | Tag every runbook step |
| Dry-run support | Surprise side effects | Add to any API the agent calls |
| Rate limits and circuit breakers | Loops and runaway retries | Cap actions per incident |
| Approval with revalidation | Approving outdated state | Re-read state at apply time |
| Staged autonomy | Trusting too early | Findings only, then proposals |
For the security side of agents more broadly, see our AI agent security playbook.
Watching the agent: why observability comes with the runbook
A traditional script fails loudly. An agent can fail quietly. AWS notes that retrofitting conventional runbooks into agentic environments creates blind spots, because agents can produce wrong outputs or loop without any exception or failed API call, and teams often lack visibility into reasoning, behavioral baselines and agent-specific playbooks.
The LegalZoom team met this directly. A real duplicate-investigation bug taught them to treat the agent as production software, with idempotency, state-awareness and concurrency discipline. If two alerts fire for the same fault, you want one investigation, not two agents each posting a different conclusion.
In practice, log every tool call, the evidence the agent cited, the decision it reached and who approved what. Then alert on the agent’s own behavior: repeated identical calls, runs that exceed a time budget, findings with no cited evidence. Our AI observability guide goes through how to set that up, and our AI incident response piece explains why agent failures need their own playbook rather than a patch on the old one.
Common mistakes with AI runbooks
- Pasting an old wiki page into a prompt. The agent follows ambiguity literally. Rewrite first.
- One giant runbook. Modular skills, one per incident type, are easier to test and to restrict.
- No owner. Systems change. Someone must review each runbook when the system it describes changes.
- Skipping the replay. If you have not run it against past incidents, you do not know what it does.
- Vibe-coded glue. The scripts and integrations the agent calls are code too. If they were generated quickly by an AI assistant and never reviewed, they become the weakest link. Our Vibe Check is a $449 one-off review aimed at exactly this situation.
Our perspective
Most teams we talk to already have the runbook. What they lack is everything around it: the identity the agent runs as, the tools it is allowed to see, the logging, the approval path, and a clear line between investigating and acting.
We build agents with that in mind. Our approach pairs AI tooling with experienced human engineers so the result is maintainable, secure and production-ready, and we stay with clients after launch because a runbook that was right in March can be wrong by June.
If your procedures touch customer data or regulated systems, local AI models are one way we support privacy and compliance, since the data stays in your environment. Start small, prove it on one incident, and widen the agent’s reach only when the logs say it has earned it.
Chad
Getting help building yours
If you have a runbook and want it running safely, we can take the integration work off your team. Our AI agent creation and workflow automation service builds agents and workflows around your procedures, and our monitoring, logging and alerts package is a $2,750 one-off. If you are unsure what your infrastructure looks like today, an Infrastructure Assessment is also $2,750 one-off, and a Senior Developer Review starts from $449. Ongoing support runs through our Ship Shape plan from $1,850 per month and our Crew plan from $2,950 per month, both listed on our code review page.

If your runbook will rely on an agent framework, our three-layer agent framework guide shows how we structure it for production.
FAQ
Is an AI runbook the same as runbook automation?
Not quite. Runbook automation executes a fixed sequence of scripted steps. An AI runbook gives an agent a procedure and lets it reason about which evidence to gather and what to propose. Many teams combine them: the agent investigates and proposes, and a scripted, tested action does the actual change after approval.
Should an AI agent be allowed to fix production on its own?
Not at the start. Google’s SRE guidance describes releasing agents at human-approved autonomy levels and scaling up through defined levels over time. Begin with read-only investigation, then proposals with human approval, and consider autonomous execution only for low-risk, well-tested steps.
How do I test an AI runbook before trusting it?
Replay past incidents with write tools disabled and compare the agent’s findings with what your responders concluded. Fix the runbook wherever they diverge. Then run it in shadow mode on live alerts, posting findings only, before it proposes any change.
What is the best first runbook to convert?
Pick a frequent incident with a known cause list and a known fix, such as a full disk or a backed-up queue. It gives you plenty of examples to replay and a low cost if the first version is wrong.
Sources
- Google Cloud: agentic AI to orchestrate security operations workflows
- OpenAI Cookbook: Build an SRE agent for incident response
- Google Cloud Security Community: Modular Security Expertise, Leveling Up AI Runbooks with Agent Skills
- Google SRE: AI engineering for reliable operations
- LegalZoom Engineering: Earning the Right to Act
- AWS re:Post: Accelerating agentic AI innovation with Unified Operations
- runbook-agent: Markdown runbooks into agent-executable steps