---
name: memory-leak-hunt
description: Confirms a memory leak under steady load, then locates it with heap snapshots and diffs in Node.js, Python tracemalloc or a JVM heap dump, and checks common sources such as unbounded caches and listeners. Use when memory grows over time or a process gets OOM-killed.
---

# Memory Leak Hunt

Memory that grows is not always a leak. Prove growth first, then find what holds the objects, then fix the holder.

## Workflow

### 1. Confirm it is a leak

- Measure resident memory over time under steady, repeated load, not a one-off spike. A leak keeps rising after warm-up and garbage collection; a cache or a buffer pool plateaus.
- Run the same operation many times (for example 10,000 requests) and sample memory at intervals.

```sh
ps -o rss= -p <pid>                       # RSS in KB
while true; do date +%T; ps -o rss= -p <pid>; sleep 10; done
docker stats --no-stream <container>
kubectl top pod <pod>
```

- Check the runtime limits: container memory limit, `--max-old-space-size`, JVM `-Xmx`. A process growing toward its limit and then restarting is the symptom you are tracing.
- Record: growth rate (MB per 1,000 requests), which operation triggers it, and whether forcing garbage collection frees anything.

### 2. Narrow the trigger

- Reproduce locally with a loop calling one endpoint or one function. Bisect by operation: which call alone makes memory climb?
- Disable features (cache, background jobs, metrics) one at a time and compare the slope.

### 3. Take snapshots and compare

Take one snapshot after warm-up, run the load, take a second one, and diff. Look at what grew in count, not at the biggest objects.

**Node.js**

```sh
node --inspect app.js                          # attach DevTools Memory tab, take 2 heap snapshots
node --heapsnapshot-signal=SIGUSR2 app.js      # kill -USR2 <pid> writes a .heapsnapshot file
```

In DevTools use "Comparison" view between snapshot 1 and 2, sort by "# Delta", and open the retainers panel for a suspicious class to see what holds it. Also check `process.memoryUsage()` (`heapUsed` vs `external` vs `arrayBuffers`): growth outside the heap points to Buffers or native addons.

**Python**

```python
import tracemalloc
tracemalloc.start(25)
snap1 = tracemalloc.take_snapshot()
# run the workload
snap2 = tracemalloc.take_snapshot()
for s in snap2.compare_to(snap1, "lineno")[:15]:
    print(s)
```

For object counts by type use `objgraph.show_growth()` if the project already has it, or `gc.get_objects()` counted by `type`. For reference cycles check `gc.garbage` and `gc.collect()`.

**JVM**

```sh
jcmd <pid> GC.heap_info
jcmd <pid> GC.class_histogram | head -30
jcmd <pid> GC.heap_dump /tmp/heap.hprof     # open in Eclipse MAT or VisualVM
```

Compare two class histograms taken minutes apart. In a heap dump, run "Leak Suspects" and follow the path to GC roots.

Heap dumps and snapshots contain real data (tokens, user content). Keep them out of the repo and do not share them without asking the user.

### 4. Check the usual suspects

- Caches and maps without a size limit or expiry, keyed by request-specific data (user id, URL, query string).
- Event listeners or subscriptions added on every request or render and never removed (`on` without `off`, `addEventListener`, observers).
- Timers and intervals never cleared; background tasks that keep references to finished requests.
- Closures capturing large objects (the whole request, response or buffer).
- Module-level or global arrays, lists and dictionaries that are only appended to; per-request data stored in globals or class attributes.
- Unclosed resources: connections, streams, file handles, cursors, child processes, database sessions.
- Logging or metrics with unbounded label cardinality (a label per user id or URL).
- Reference cycles with finalizers, thread-local data in thread pools, string interning of unique values.

### 5. Fix and verify

- Fix the holder: bound the cache (max size, TTL, LRU), remove the listener on teardown, close the resource in a finally block.
- Re-run the same load and compare the memory slope with the first measurement. The curve must flatten.
- Add a guard where possible: a test that runs the operation many times and asserts bounded size of the suspect structure.

## Report format

- Evidence of the leak: measurements before and after (numbers, duration, load).
- What grows and what retains it (class or call site, retainer path).
- The fix and the new measurement.
- Remaining risk: anything not confirmed in production.

## Pitfalls

- A single snapshot rarely tells you anything; always diff two.
- Do not raise memory limits or restart on a schedule as the final answer; mention it only as a stopgap and say so.
- Profilers slow the process and change timing; do not use their numbers as production performance figures.
- Do not attach a profiler or take dumps in production without asking the user; dumps pause the process and can contain sensitive data.
- RSS may not shrink after GC because the allocator keeps pages; compare heap used and RSS together.

Source: [AnyModel skills catalog](https://anymodel.org/en/skills) · License: [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/)
