Observability and instrumentation

Instruments a feature with structured JSON logs and request IDs, RED and USE metrics with bounded labels, OpenTelemetry traces and symptom-based alerts with runbooks, then tests the telemetry. Use when a feature, job or integration is headed for production.

DevOpsPack: Agent Skills by Addy Osmani238 linesobservability-and-instrumentation
Author
Source
addyosmani/agent-skills/skills/observability-and-instrumentation @ 8c99fa3
License
MIT · Copyright (c) 2025 Addy Osmani

About this skill

What it does

The skill has the agent write telemetry with the feature, not after the first incident. It starts with two to four questions an on-call engineer will ask about the feature; every log, metric and span must answer one of them. Metrics say that something is wrong, traces say where, logs say why.

Logs are JSON events with a stable name and fields, never interpolated strings, with consistent levels and a request ID on every line; when a scheduler, a replay endpoint and a CLI share one log, each run also carries an entryPoint field. Metrics follow RED for endpoints and dependencies and USE for queues and pools, latency goes into histograms read at p95/p99, and labels come only from small fixed sets — never user IDs or raw URLs. Tracing uses OpenTelemetry auto-instrumentation plus a few manual spans, with context carried across queues. Alerts fire on symptoms users feel, have two severities (page and ticket) and each links to a short runbook in docs/runbooks/. Finally the agent tests the telemetry itself: it forces an error in staging, finds it by request ID and fires each new alert once.

The folder holds SKILL.md and the licence. The checklist it mentions, ../../references/observability-checklist.md, lives in the author's repository and is not installed with the skill.

When to use it, and when not

Use it for any feature, endpoint, background job or integration headed for production, when reviewing a PR that adds retries, queues or cross-service calls, and after an incident nobody could explain from the data.

Not for a failure happening right now, profiling measured slowness or launch-day rollback triggers: the skill sends those to the skills listed below.

How to run it

The install command below puts SKILL.md and the licence in place. The agent picks the skill up by its description when you add logging, metrics, tracing or alerts; to call it yourself:

  • Claude Code: /observability-and-instrumentation, or ask for the skill by name.
  • Codex: $observability-and-instrumentation.
  • Cursor: type / in the Agent chat and pick observability-and-instrumentation.

Example: /observability-and-instrumentation Instrument the new webhook consumer: logs, RED metrics and one alert with a runbook.

It hands off to Debugging and error recovery, Performance optimization and Shipping and launch, and takes the no-secrets-in-logs rule from Security hardening.

How it differs from similar skills

Log triage and Incident runbook start after something broke: one reads existing logs to find the first failure, the other walks a live incident to a postmortem. This skill decides beforehand what those logs, metrics and alerts contain. Shipping and launch lists what to watch during a rollout; the signals it watches are built here. Sentry issues and events only reads errors that have already reached Sentry, through its CLI; this skill decides what telemetry the code emits in the first place.

Install

Run one command — it saves SKILL.md and the files it uses into your agent's skills folder. Claude Code and Codex pick up a new skill without a restart; if it does not show up, restart the agent.

Claude Code — macOS, Linux
curl -fsSL --create-dirs -o ~/.claude/skills/observability-and-instrumentation/SKILL.md https://anymodel.org/en/skills/addy-osmani/observability-and-instrumentation/SKILL.md -o ~/.claude/skills/observability-and-instrumentation/LICENSE https://anymodel.org/en/skills/addy-osmani/observability-and-instrumentation/LICENSE
Codex, Cursor — macOS, Linux
curl -fsSL --create-dirs -o ~/.agents/skills/observability-and-instrumentation/SKILL.md https://anymodel.org/en/skills/addy-osmani/observability-and-instrumentation/SKILL.md -o ~/.agents/skills/observability-and-instrumentation/LICENSE https://anymodel.org/en/skills/addy-osmani/observability-and-instrumentation/LICENSE
Claude Code — Windows PowerShell
foreach ($f in 'SKILL.md','LICENSE') { $p = Join-Path "$HOME\.claude\skills\observability-and-instrumentation" $f; New-Item -ItemType Directory -Force (Split-Path $p) | Out-Null; Invoke-WebRequest "https://anymodel.org/en/skills/addy-osmani/observability-and-instrumentation/$f" -UseBasicParsing -OutFile $p }
Codex, Cursor — Windows PowerShell
foreach ($f in 'SKILL.md','LICENSE') { $p = Join-Path "$HOME\.agents\skills\observability-and-instrumentation" $f; New-Item -ItemType Directory -Force (Split-Path $p) | Out-Null; Invoke-WebRequest "https://anymodel.org/en/skills/addy-osmani/observability-and-instrumentation/$f" -UseBasicParsing -OutFile $p }

Any other agent: save the file into its skills folder, or give it the link to the file: https://anymodel.org/en/skills/addy-osmani/observability-and-instrumentation/SKILL.md

SKILL.md

Raw

Observability and Instrumentation

Overview

Code you can't observe is code you can't operate. Observability is the ability to answer "what is the system doing and why?" from the outside, using the telemetry the code emits. Instrumentation is not a post-launch add-on — it's written alongside the feature, the same way tests are. If a feature ships without telemetry, the first user-reported bug becomes archaeology instead of a query.

When to Use

  • Building any feature that will run in production
  • Adding a new service, endpoint, background job, or external integration
  • A production incident took too long to diagnose ("we couldn't tell what happened")
  • Setting up or reviewing alerting rules
  • Reviewing a PR that adds I/O, retries, queues, or cross-service calls

NOT for:

  • Diagnosing a failure happening right now — use the debugging-and-error-recovery skill (observability is what makes that skill fast next time)
  • Profiling and optimizing measured slowness — use the performance-optimization skill
  • Launch-day monitoring checklists and rollback triggers — see the shipping-and-launch skill; this skill covers the instrumentation that feeds them

Process

1. Define "working" before instrumenting

Telemetry without a question is noise. Before adding any instrumentation, write down 2–4 questions an on-call engineer will ask about this feature:

FEATURE: checkout payment retry
QUESTIONS ON-CALL WILL ASK:
1. What fraction of payments succeed on first attempt vs after retry?
2. When a payment fails permanently, why? (provider error? timeout? validation?)
3. Is the payment provider slower than usual?
→ Every signal below must help answer one of these.

If you can't name the questions, you're not ready to instrument — you'll log everything and learn nothing.

2. Pick the right signal for each question

Signal Answers Cost profile Example
Structured log "What happened in this specific case?" Per-event; grows with traffic payment_failed with provider error code
Metric "How often / how fast, in aggregate?" Fixed per series; cheap to query p99 latency of provider calls
Trace "Where did time go across services?" Per-request; usually sampled One slow checkout, broken down by hop

Rule of thumb: metrics tell you that something is wrong, traces tell you where, logs tell you why.

3. Structured logging

Log events, not prose. Every log line is a JSON object with a stable event name and machine-readable fields:

// BAD: string interpolation — unqueryable, inconsistent
logger.info(`Payment ${id} failed for user ${userId} after ${n} retries`);

// GOOD: stable event name + structured fields
logger.warn({
  event: 'payment_failed',
  paymentId: id,
  provider: 'stripe',
  errorCode: err.code,
  attempt: n,
}, 'payment failed');

Log levels — use them consistently:

Level Meaning On-call action
error Invariant broken; someone may need to act Investigate
warn Degraded but handled (retry succeeded, fallback used) Watch for trends
info Significant business event (order placed, job finished) None
debug Diagnostic detail Off in production by default

Correlation IDs are mandatory. Generate (or accept) a request ID at the system boundary and attach it to every log line, span, and outbound call. Without it, you cannot reconstruct a single request from interleaved logs:

// Express: child logger per request, ID propagated downstream
app.use((req, res, next) => {
  req.id = req.headers['x-request-id'] ?? crypto.randomUUID();
  req.log = logger.child({ requestId: req.id });
  res.setHeader('x-request-id', req.id);
  next();
});

When several entry points write to one log, name the entry point. A correlation ID identifies a run; it does not say which code path started it. The same job reached by a scheduler, by a replay endpoint, and by a manual CLI run produces interchangeable lines in one sink, so attributing a line falls back to elimination — cross-reading the scheduler's history, the process table, a deploy log — and that argument holds only as long as those external records happen to still exist. Stamp the entry point where the run starts, next to the correlation ID, and propagate both the same way:

// One helper for every entry point: the run's own logger carries both fields.
// `entryPoint`, not `source` — ECS reserves `source.*` for network fields.
export const runLog = (entryPoint: 'scheduler' | 'replay_endpoint' | 'cli', runId: string) =>
  logger.child({ entryPoint, requestId: runId });

// scheduler tick        -> runLog('scheduler', crypto.randomUUID())
// POST /jobs/:id/replay -> runLog('replay_endpoint', req.id)
// CLI invocation        -> runLog('cli', process.env.RUN_ID ?? crypto.randomUUID())

Both fields have to cross the same boundaries as the correlation ID — queue metadata, HTTP headers — or a worker re-derives the entry point and guesses. A field that merely correlates with an entry point is a hint, not an attribution: anything that can invoke the job can reproduce it.

Never log secrets, tokens, passwords, or full PII. This is a hard rule from the security-and-hardening skill — telemetry pipelines are a classic data-leak path. Allowlist fields; don't log whole request bodies.

4. Metrics

For request-driven services, instrument RED on every endpoint and every external dependency: Rate (requests/sec), Errors (failure rate), Duration (latency histogram, not average). For resources (queues, pools, hosts), use USE: Utilization, Saturation, Errors.

As with tracing, the vendor-neutral path is the OpenTelemetry metrics API (same SDK and context as step 5). The example below uses Prometheus' prom-client — one common backend choice, not the only one; the RED/USE and cardinality rules are identical either way.

import { Histogram } from 'prom-client';

const httpDuration = new Histogram({
  name: 'http_request_duration_seconds',
  help: 'HTTP request duration',
  labelNames: ['method', 'route', 'status_class'],  // '2xx', not '200'
  buckets: [0.05, 0.1, 0.25, 0.5, 1, 2.5, 5],
});

Cardinality is the failure mode. Every unique label combination is a separate time series. Labels must come from small, fixed sets (route template, status class, provider name). Never use user IDs, raw URLs, error messages, or other unbounded values as labels — that belongs in logs and traces.

OK as label:    route="/api/tasks/:id"   status_class="5xx"   provider="stripe"
NEVER a label:  user_id, email, request_id, full URL, error message text

Track averages never, percentiles always: an average hides the 1% of users having a terrible time. Use histograms and read p50/p95/p99.

5. Distributed tracing

Use OpenTelemetry — it's the vendor-neutral standard, and auto-instrumentation covers HTTP, gRPC, and common DB clients with near-zero code:

// tracing.ts — must be imported before anything else
import { NodeSDK } from '@opentelemetry/sdk-node';
import { getNodeAutoInstrumentations } from '@opentelemetry/auto-instrumentations-node';

const sdk = new NodeSDK({
  serviceName: 'checkout-service',
  instrumentations: [getNodeAutoInstrumentations()],
});
sdk.start();

Add manual spans only around meaningful internal units of work (e.g., applyDiscounts, chargeProvider) and attach the attributes on-call will filter by. Propagate context across every async boundary — HTTP headers, queue message metadata — or the trace dies at the gap. Sample head-based at a low rate by default; keep 100% of errors if your backend supports tail sampling.

6. Alerting

Alert on symptoms users feel, not on causes:

SYMPTOM (page-worthy):           CAUSE (dashboard, not a page):
error rate > 1% for 5 min        CPU at 85%
p99 latency > 2s                 one pod restarted
queue age > 10 min               disk at 70%

Cause-based alerts fire when nothing is wrong and miss failures you didn't predict. Symptom-based alerts fire exactly when users are hurt, regardless of the cause.

Rules for every alert you create:

  1. It must be actionable. If the response is "ignore it, it self-heals", delete the alert.
  2. It links to a runbook — even three lines: what it means, first query to run, escalation path.
  3. It has a threshold and duration justified by the SLO or by historical data, not by a guess.
  4. Use two severities only: page (user-facing, act now) and ticket (degradation, act this week). A third tier becomes noise that trains people to ignore everything.
Writing Runbooks

Rule 2 above requires every alert to link to a runbook. A runbook's job is to answer three questions without requiring the reader to think: what is happening, what to check first, and who to call if that doesn't resolve it. Store in docs/runbooks/ named after the alert.

Minimum viable runbook (three lines):

# Runbook: High Error Rate on /api/tasks
**Means:** DB connection pool likely exhausted, or a bad deploy.
**First check:** `SELECT count(*) FROM pg_stat_activity WHERE backend_type = 'client backend';`
  — if count > pool limit, see Step 2. (Swap in the equivalent for your database.)
**Escalate to:** #db-oncall or engineering on-call rotation.

When to expand beyond three lines: add steps only when the first check alone isn't enough to decide. A five-step runbook that covers the three most common causes is better than a twenty-step document that covers every edge case and gets skimmed.

Keep runbooks current. Update the runbook as part of closing every incident it was used in — a stale runbook builds false confidence. If a step was wrong or missing, fix it before marking the incident resolved.

7. Verify the telemetry itself

Instrumentation is code; it can be wrong. Before calling the work done, trigger the paths and look at the actual output:

  • Force an error in staging → find it in the logs by requestId, confirm fields are structured (not [object Object])
  • Send test traffic → confirm metric series appear with the expected labels and sane values
  • Follow one request across services in the tracing UI → no broken spans
  • Fire each new alert once (lower the threshold temporarily) → confirm it reaches the right channel and the runbook link works

Common Rationalizations

Rationalization Reality
"I'll add logging after it works" "After" becomes "after the first incident", which is the most expensive moment to discover you're blind. Instrument as you build.
"More logs = more observability" Unstructured noise makes incidents slower, not faster. Three queryable events beat three hundred prose lines.
"console.log is fine for now" Unstructured output can't be filtered, correlated, or alerted on. The structured logger costs five extra minutes once.
"We can just look at the dashboards when something breaks" Dashboards built without defined questions show you everything except the answer. Start from on-call questions.
"Alert on everything important, we'll tune later" A noisy pager trains people to ignore it. The tuning never happens; the missed real page does.
"User ID as a metric label makes debugging easier" It also makes your metrics backend fall over. High-cardinality lookups belong in logs and traces.
"Tracing is overkill for our two services" Two services already means cross-service latency questions logs can't answer. Auto-instrumentation makes the cost trivial.

Red Flags

  • A feature PR with retries, queues, or external calls and zero new telemetry
  • Log lines built by string interpolation instead of structured fields
  • No correlation/request ID — each log line is an orphan
  • One log stream fed by a scheduler, a webhook, and manual runs, with no field naming which one produced the line
  • Metrics labeled with user IDs, raw URLs, or error message text (cardinality bomb)
  • Latency tracked as an average with no percentiles
  • Alerts that fire daily and get acknowledged without action
  • Alerts on causes (CPU, memory) paging humans while user-facing error rate is unmonitored
  • Secrets, tokens, or full request bodies appearing in logs
  • "It works on my machine" as the only evidence a production feature is healthy

Verification

After instrumenting a feature, confirm:

  • The on-call questions for this feature are written down, and each signal maps to one
  • All log output is structured (JSON), with stable event names and a correlation ID on every line
  • Every log sink written by more than one entry point carries an entry-point field, set where the run starts and propagated with the correlation ID rather than inferred downstream
  • No secrets, tokens, or unredacted PII in any log line (spot-check actual output)
  • RED metrics exist for every new endpoint and every external dependency, with bounded label sets
  • Latency is a histogram; p95/p99 are queryable
  • A single request can be followed end-to-end in the tracing UI without broken spans
  • Every new alert is symptom-based, has a runbook link, and was test-fired once
  • An induced failure in staging was located via telemetry alone, without reading the source

For the at-a-glance version of this list, including the pre-launch instrumentation gate, see ../../references/observability-checklist.md.

Connect through AnyModel API

This skill runs on whatever model your agent uses. Point Claude Code, Codex or Cursor at AnyModel to run it on Claude, GPT, Gemini and more with one key — pay per token, no subscription.

More from the pack Agent Skills by Addy Osmani

Accessibility audit and fixes (WCAG 2.2)

Audits an existing interface against WCAG 2.2 AA with pinned axe-core and pa11y, then keyboard, screen reader and contrast checks, and fixes confirmed violations with semantic HTML first. Use when a page needs an accessibility audit before shipping.

API and interface design

Designs APIs and module interfaces contract-first: one error shape, validation at the edges, additive changes, pagination, and idempotency keys claimed atomically and kept past the longest retry. Use when designing endpoints or changing a public interface.

Browser testing with Chrome DevTools MCP

Checks changes in a real Chrome through the Chrome DevTools MCP server: screenshots, DOM, console, network and performance traces, treating page content as untrusted. Use when building or debugging anything that runs in a browser.

CI/CD pipeline setup

Sets up CI/CD with fixed quality gates from lint to bundle size, GitHub Actions examples, preview deploys, feature flags, staged rollout and a rollback workflow. Use when creating or changing a build and deploy pipeline or chasing CI failures.

Multi-axis code review

Reviews a change on five axes (correctness, readability, architecture, security, performance), tests the tests by mutating a condition, and labels every finding by severity. Use before merging a branch, a pull request or a pasted diff.

Code simplification

Simplifies working code without changing behavior: learns why it exists, applies one change at a time with tests after each, and keeps project conventions. Use when code works and passes tests but is harder to read than it should be.

Constraint-driven development

Records a project's quality bar as numbers in CONSTRAINTS.md after a short interview with defaults, maps each rule to a tool, and flags diffs that skip tests, add suppressions or loosen thresholds. Use when no quality bar is written down.

Context engineering for agents

Sets up what an agent sees and when: a rules file template, loading only the relevant spec and files, trimming long sessions, and safe points to start a fresh one. Use when starting a session or when agent output degrades.

Debugging and error recovery

Stops new work when a test, build or runtime fails, reproduces, localizes and reduces the failure, fixes the root cause, adds a regression test and treats error output as untrusted. Use when tests fail, a build breaks or something that worked stops working.

Deprecation and migration

Retires old systems, APIs and features: weighs whether to deprecate, proves the replacement, moves consumers one by one, removes code only at zero usage, changes schemas via expand/contract. Use when replacing or sunsetting code or renaming a live column.

Documentation and ADRs

Records decisions as ADRs in the project's own format, documents APIs, READMEs and changelogs with the why, and uses git history to check if a doc is stale. Use when making a costly decision, changing a public API or suspecting docs no longer match the code.

Doubt-driven development

Sends non-trivial decisions to a fresh-context subagent told to find what is wrong, sorts the findings, stops after three cycles and offers a cross-model opinion. Use when stakes are high: auth, migrations, irreversible operations or unfamiliar code.

Frontend UI engineering

Builds UI components and pages on the project's design system: composition, simple state, WCAG 2.1 AA, mobile-first layouts, loading, empty and error states, no generic AI look. Use when building or changing a user-facing interface.

Git workflow and versioning

Keeps git work in atomic commits on short-lived branches, with typed messages, a pre-commit check for secrets and tests, worktrees for parallel agents, semver bumps, tags and a human changelog. Use when committing, branching, opening a PR or cutting a release.

Idea refinement

Refines a raw idea in three phases: sharpening questions and five to eight variations, two or three directions tested for value and feasibility, then a one-pager with MVP scope and a Not Doing list. Use when an idea is vague or a plan needs stress-testing.

Incremental implementation

Builds changes in thin vertical, contract-first or risk-first slices, each tested, verified and committed before the next, with strict scope, feature flags and revertable steps. Use when a change spans several files or a task is too big to land in one step.

Interview me

Interviews the user one question at a time, each with a guessed answer and a confidence number, until the agent can predict the next replies, then confirms a statement of intent before any spec or code. Use when a request leaves who, why or success unclear.

Performance optimization

Optimizes by measurement: takes a baseline, fixes the one proven bottleneck, re-measures against noise and reverts neutral changes, with worked fixes for N+1 queries, indexes, caching and bundles. Use when something is slow or misses a performance budget.

Planning and task breakdown

Breaks a spec into small vertical-slice tasks with acceptance criteria, verification steps, sizes, dependency order and checkpoints, saved to tasks/todo.md or a named tracker. Use when a spec is ready but the work is too big to start in one go.

Security hardening

Hardens code with a short threat model, always, ask-first and never rules, and patterns for injection, XSS, access control, SSRF, secrets, dependencies, personal data and LLM output. Use when code handles user input, auth, sensitive data or outside services.

Shipping and launch

Prepares a production launch with a pre-launch checklist, a feature-flag rollout from 5% to 100%, thresholds for advancing or rolling back, an error-budget gate, a rollback plan and first-hour checks. Use when a feature or major change is about to ship.

Source-driven development

Grounds framework code in official docs: reads exact versions from the dependency file, fetches the relevant page, follows the documented API and cites full URLs, flagging anything unverified. Use when framework-specific code must match the version in use.

Spec-driven development

Writes a spec before code, surfacing assumptions and covering objective, commands, structure, style, tests and boundaries, then gates plan, tasks and code on approval, with capability maps and delta specs. Use when new work has vague requirements.

Test-driven development, red-green-refactor

Drives changes with red-green-refactor: finds the repository's own test commands, writes the failing test first, reproduces each bug with a test before fixing it, and sets rules on mocks and naming. Use when adding logic, fixing a bug or changing behavior.