Разбор логов

Читает логи и стектрейсы, находит первую настоящую ошибку, сопоставляет события по request id и времени, отделяет причину от каскада последующих ошибок, с рецептами grep и jq. Для расследования ошибок и инцидентов.

ОтладкаПак: Отладка100 строкlog-triage

Установка

Выполните одну команду — она сохранит SKILL.md в папку скиллов агента. Перезапустите агента, и он подхватит скилл сам.

Claude Code — macOS, Linux
mkdir -p ~/.claude/skills/log-triage && curl -fsSL https://anymodel.org/en/skills/debugging/log-triage/SKILL.md -o ~/.claude/skills/log-triage/SKILL.md
Codex, Cursor — macOS, Linux
mkdir -p ~/.agents/skills/log-triage && curl -fsSL https://anymodel.org/en/skills/debugging/log-triage/SKILL.md -o ~/.agents/skills/log-triage/SKILL.md
Claude Code — Windows PowerShell
New-Item -ItemType Directory -Force "$HOME\.claude\skills\log-triage" | Out-Null; Invoke-WebRequest https://anymodel.org/en/skills/debugging/log-triage/SKILL.md -UseBasicParsing -OutFile "$HOME\.claude\skills\log-triage\SKILL.md"
Codex, Cursor — Windows PowerShell
New-Item -ItemType Directory -Force "$HOME\.agents\skills\log-triage" | Out-Null; Invoke-WebRequest https://anymodel.org/en/skills/debugging/log-triage/SKILL.md -UseBasicParsing -OutFile "$HOME\.agents\skills\log-triage\SKILL.md"

Другой агент: сохраните файл в его папку скиллов или дайте ему ссылку на файл: https://anymodel.org/en/skills/debugging/log-triage/SKILL.md

SKILL.md

Исходник

Log Triage

The loudest error in a log is rarely the first one. Your goal is to find the earliest event that went wrong and show how the later errors follow from it.

Workflow

1. Define the window

  • Get the time range of the problem, the affected service, and any identifier the user has: request id, user id, order id, job id, pod name.
  • Note the timezone of the logs (UTC vs local) before comparing times with anything else.
  • Find out where logs live and how big they are before opening them. Never print a multi-gigabyte file; count and filter first.
ls -lh /var/log/app/
wc -l app.log
kubectl logs -l app=api --all-containers --prefix --tail=-1 --since=30m --timestamps > api.log   # all pods; deploy/api reads only one
docker logs --since 30m --timestamps api > api.log 2>&1
journalctl -u api --since "2026-11-11 10:00" --until "2026-11-11 10:30" > api.log

Save the slice to a working file outside the repo so you can grep it repeatedly.

2. Get the shape of the problem

Count before you read:

grep -c ERROR api.log
grep -E 'ERROR|FATAL|Exception|panic' api.log | sed -E 's/[0-9a-f-]{8,}/<id>/g; s/[0-9]+/N/g' | sort | uniq -c | sort -rn | head -20

The normalized message list tells you which error classes exist and how often. Many lines of one class usually means one cause repeating.

For JSON logs:

jq -r 'select(.level=="error") | [.time,.msg] | @tsv' api.log | head -50
jq -r '.level' api.log | sort | uniq -c
jq -c 'select(.request_id=="abc123")' api.log

3. Find the FIRST failure

  • Sort by time and read the first error or warning in the window, then look at the 20 lines before it. The trigger is often a warning, a timeout, a restart or a config reload, not an error.
  • Check the moments around process starts: grep -nE 'starting|listening|shutdown|SIGTERM|OOM|killed' api.log.
  • If multiple services are involved, merge their logs by timestamp and find which one failed first.
sort -m -k1,1 svc-a.log svc-b.log | less

4. Separate cause from cascade

Typical cascade patterns:

  • A dependency fails (database, cache, upstream API), then every caller logs timeouts and 5xx errors.
  • One crash leads to restarts, then connection-refused errors from peers.
  • One retry storm multiplies the same error many times.

Ask for each error class: did it appear before or after the first failure, and could it be a consequence of it? Errors that start only after the first failure are noise until proven otherwise.

5. Correlate

  • Follow one failing request end to end with its id across services: grep -h 'abc123' *.log | sort.
  • If there is no request id, correlate by timestamp within a small window plus user or resource id, and say that the match is approximate.
  • Compare a failing request with a successful one of the same type and list the differences.

6. Read the stack trace properly

  • Find the deepest frame in project code, then the frame that called it with the bad data.
  • For "caused by" chains, the last cause is the root; the first is the wrapper.
  • Check the line numbers against the deployed version, not the working tree: git log and the running build may differ.

7. Confirm with data

State your hypothesis and check it against the logs: does the count match, does the timing match, do all affected requests share the property you claim? A cause that explains only some of the errors is incomplete.

Report format

  • Timeline: 3 to 8 lines, time, event, source.
  • First failure and the evidence line (quote one or two log lines, redacted).
  • Cascade: which errors are consequences and why.
  • Impact: how many requests or users, over what period.
  • Next step: fix, more logging, or what to check if the cause is still uncertain.

Pitfalls

  • Never paste tokens, cookies, emails or personal data from logs into reports. Redact them.
  • Do not call a cause confirmed from correlation of time alone.
  • Log timestamps can be out of order across hosts because of clock drift; do not over-trust sub-second ordering.
  • grep on rotated logs misses .1, .gz files; use zgrep for those.
  • Sampled or rate-limited logging hides repeats; a missing line is not proof the event did not happen.
  • Do not truncate or delete production logs to make them easier to read.

Source: AnyModel skills catalog · License: CC BY 4.0

Подключить через AnyModel API

Скилл работает на той модели, которую использует агент. Подключите Claude Code, Codex или Cursor к AnyModel и запускайте его на Claude, GPT, Gemini и других моделях по одному ключу — оплата за токены, без подписки.

Ещё из пака Отладка

Поиск первопричины бага

Отлаживает баг по шагам: воспроизводит, изолирует, проверяет по одной гипотезе с доказательствами, чинит причину, а не симптом, и добавляет регрессионный тест. Для любых падений и неожиданного поведения.

Поиск регрессии через git bisect

Находит коммит, который внёс регрессию, через git bisect вручную и автоматически с git bisect run, пропускает нестабильные и несобираемые коммиты. Для случаев, когда раньше работало, а теперь сломано.

Поиск утечек памяти

Подтверждает утечку под стабильной нагрузкой и ищет её через heap snapshot и сравнение в Node.js, tracemalloc в Python или heap dump в JVM, проверяет частые источники: кэши без лимита и слушатели. Используйте, когда память растёт со временем или процесс убивает OOM.

Охота на нестабильные тесты

Находит, почему тест то проходит, то падает: прогоны в цикле, случайный порядок, тайминги, общее состояние, часовые пояса, сеть, порядок в коллекциях, и чинит причину. Используйте, когда CI нестабилен или тест падает лишь иногда.