Incident runbook

Guides triage of a production incident: assess impact, stabilize first by rollback or feature flag, keep a timeline, communicate, then write a blameless postmortem. Use when the user reports an outage, errors in production, a failed deploy or asks for a postmortem.

DevOpsPack: DevOps106 linesincident-runbook

Install

Run one command — it saves SKILL.md into your agent's skills folder. Restart the agent and the skill is picked up automatically.

Claude Code — macOS, Linux
mkdir -p ~/.claude/skills/incident-runbook && curl -fsSL https://anymodel.org/en/skills/devops/incident-runbook/SKILL.md -o ~/.claude/skills/incident-runbook/SKILL.md
Codex, Cursor — macOS, Linux
mkdir -p ~/.agents/skills/incident-runbook && curl -fsSL https://anymodel.org/en/skills/devops/incident-runbook/SKILL.md -o ~/.agents/skills/incident-runbook/SKILL.md
Claude Code — Windows PowerShell
New-Item -ItemType Directory -Force "$HOME\.claude\skills\incident-runbook" | Out-Null; Invoke-WebRequest https://anymodel.org/en/skills/devops/incident-runbook/SKILL.md -UseBasicParsing -OutFile "$HOME\.claude\skills\incident-runbook\SKILL.md"
Codex, Cursor — Windows PowerShell
New-Item -ItemType Directory -Force "$HOME\.agents\skills\incident-runbook" | Out-Null; Invoke-WebRequest https://anymodel.org/en/skills/devops/incident-runbook/SKILL.md -UseBasicParsing -OutFile "$HOME\.agents\skills\incident-runbook\SKILL.md"

Any other agent: save the file into its skills folder, or give it the link to the file: https://anymodel.org/en/skills/devops/incident-runbook/SKILL.md

SKILL.md

Raw

Incident Runbook

Help the user handle a live production problem calmly and in the right order: impact, stabilize, diagnose, communicate, learn. You assist; the user owns decisions and any action on production.

Hard rules

  • Never run a destructive or state-changing command on production (restart, scale, rollback, delete, migration, config change, DB write) without the user's explicit confirmation of that specific command. Propose it, say what it does and how to undo it, then wait.
  • Prefer read-only investigation first: logs, metrics, git log, deploy history, status pages.
  • Do not print secrets found in logs or configs. Mask them in your notes.
  • Stabilize before root cause. A working rollback beats an elegant diagnosis.

Step 1: Assess impact (first 5 minutes)

Ask or determine, and write down:

  • What is broken, for whom, since when (first alert, first user report, last deploy).
  • Severity: full outage, partial degradation, data risk, security exposure.
  • Blast radius: all users, one region, one tenant, one endpoint.
  • Is data being lost or corrupted right now? If yes, that outranks everything; propose stopping the writer.

Step 2: Find what changed

git log --since="6 hours ago" --oneline origin/main
kubectl rollout history deployment/<name> -n <ns>     # if Kubernetes
gh run list --workflow deploy --limit 5                # recent deploys

Look for: deploys, config or env changes, feature flag flips, migrations, certificate or credential expiry, dependency or provider outages, traffic spikes, full disks, exhausted connection pools.

Step 3: Stabilize

Choose the least risky mitigation that restores service, in this order:

  1. Turn off the feature flag of the suspect change.
  2. Roll back to the last known good version.
  3. Scale out, restart, or fail over if the cause is capacity or a stuck process.
  4. Shed load, add a rate limit, serve a degraded response.
  5. Forward-fix only if rollback is impossible (for example, an irreversible migration) and the fix is tiny and tested.

Before proposing a rollback, check whether a database migration shipped with the release and whether the old code can still read the new schema. State that risk to the user.

After each mitigation, verify with the same signal that raised the alarm (error rate, latency, health endpoint) and report whether it recovered.

Step 4: Keep a timeline

Maintain a running list with timestamps (state the timezone, UTC preferred):

14:02 alert: 5xx rate 12% on /checkout
14:05 impact confirmed: ~30% of checkouts failing, EU only
14:09 suspect: deploy 14:00 changed payment client timeout
14:12 user approved rollback to previous release
14:15 5xx back to baseline

Record decisions and who made them, not just events.

Step 5: Communicate

Draft short updates for the user to send (never send on your own):

  • Initial: what is affected, that it is being investigated, time of next update.
  • Progress: current hypothesis or mitigation, no speculation presented as fact.
  • Resolved: what was restored, whether any follow-up risk remains. Keep a fixed update cadence (for example every 30 minutes) even when there is no news.

Step 6: Find the root cause (after stability)

  • Reproduce in a non-production environment or from logs; follow the chain of "why" until you reach a process or design cause, not a person.
  • Separate trigger (what happened) from contributing factors (why it was not caught: missing alert, no canary, no test).

Step 7: Blameless postmortem

Produce this template filled with facts from the timeline:

# Postmortem: <title>
Date / duration / severity:
Summary: (2-3 sentences, plain language)
Impact: users, requests, revenue or data affected
Timeline: (from the running list)
Root cause:
Trigger vs contributing factors:
What went well:
What went poorly:
Where we got lucky:
Action items: | action | owner | priority | due |

Action items must be concrete (add alert on X, add a canary step, add test Y). Use blameless language: describe systems and decisions, never blame individuals.

Output format

During the incident reply with: current status, last verified signal, next proposed step (with the exact command and its risk), and what you need from the user. After resolution deliver the timeline and the postmortem draft.

Pitfalls

  • Do not start debugging code while users are still failing and a rollback is available.
  • Do not change several things at once; you will not know what helped.
  • Do not delete logs or evidence while cleaning up.
  • Do not close the incident until the metrics have stayed normal for a reasonable window.

Source: AnyModel skills catalog · License: CC BY 4.0

Connect through AnyModel API

This skill runs on whatever model your agent uses. Point Claude Code, Codex or Cursor at AnyModel to run it on Claude, GPT, Gemini and more with one key — pay per token, no subscription.

More from the pack DevOps